![1
C H A P T E R 1
Introduction
Today’s datacenters have emerged from the collection of loosely connected workstations, which
shaped the humble beginnings of the Internet, and grown into massive “warehouse-scale comput-
ers” (Figure 1.1) capable of running the most demanding workloads.Barroso and Hölzle describe the
architecture of a warehouse-scale computer (WSC) [9] and give an overview of the programming
model and common workloads executed on these machines.The hardware building blocks are pack-
aged into “racks” of about 40 servers, and many racks are interconnected using a high-performance
network to form a “cluster” with hundreds or thousands of tightly-coupled servers for performance,
cooling
towers
power substation
warehouse-scale
computer
Figure 1.1: A datacenter with cooling infrastructure and power delivery highlighted.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-21-320.jpg)
![1.1. FROM SUPERCOMPUTING TO CLOUD COMPUTING 3
1.1 FROM SUPERCOMPUTING TO CLOUD COMPUTING
As the ARPANET transformed into the Internet over the past forty years, and the World Wide
Web emerges from adolescence and turns twenty, this metamorphosis has seen changes in both
supercomputing and cloud computing. The supercomputing industry was born in 1976 when Sey-
mour Cray announced the Cray-1 [54]. Among the many innovations were its processor design,
process technology, system packaging, and instruction set architecture. The foundation of the ar-
chitecture was based on the notion of vector operations that allowed a single instruction to operate
on an array, or “vector,” of elements simultaneously. In contrast to scalar processors of the time
whose instructions operated on single data items. The vector parallelism approach dominated the
high-performance computing landscape for much of the 1980s and early 1990s until “commodity”
microprocessors began aggressively implementing forms of instruction-level parallelism (ILP) and
better cache memory systems to exploit spatial and temporal locality exhibited by most applications.
Improvements in CMOS process technology and full-custom CMOS design practices allowed mi-
croprocessors to quickly ramp up clock rates to several gigahertz. This coupled with multi-issue
pipelines; efficient branch prediction and speculation eventually allowed microprocessors to catch
up with their proprietary vector processors from Cray, Convex, and NEC. Over time, conventional
microprocessors incorporated short vector units (e.g., SSE, MMX, AltiVec) into the instruction set.
However, the largest beneficiary of vector processing has been multimedia applications as evidenced
by the jointly developed (by Sony,Toshiba,and IBM) Cell processor which found widespread success
in Sony’s Playstation3 game console, and even some special-purpose computer systems like Mercury
Systems.
Parallel applications eventually have to synchronize and communicate among parallel threads.
Amdahl’s Law is relentless and unless enough parallelism is exposed,the time spent orchestrating the
parallelism and executing the sequential region will ultimately limit the application performance [27].
1.2 BEOWULF: THE CLUSTER IS BORN
In 1994 Thomas Sterling (then dually affiliated with the California Institute of Technology and
NASAs JPL) and Donald Becker (then a researcher at NASA) assembled a parallel computer that
became known as a Beowulf cluster1. What was unique about Beowulf [61] systems was that they
were built from common “off-the-shelf” computers, as Figure 1.3 shows, system packaging was not
an emphasis. More importantly, as a loosely-coupled distributed memory machine, Beowulf forced
researchers to think about how to efficiently program parallel computers. As a result, we benefited
from portable and free programming interfaces such as parallel virtual machines (PVM), message
passing interfaces (MPICH and OpenMPI), local area multiprocessor (LAM); with MPI being
embraced by the HPC community and highly optimized.
The Beowulf cluster was organized so that one machine was designated the “server,” and it
managed job scheduling, pushing binaries to clients, and monitoring. It also acted as the gateway
1The genesis of the name comes from the poem which describes Beowulf as having “thirty men’s heft of grasp in the gripe of his
hand.”](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-23-320.jpg)
![4 1. INTRODUCTION
Figure 1.3: An 128 processor Beowulf cluster at NASA.
to the “outside world,” so researchers had a login host. The model is still quite common: with some
nodes being designated as service and IO nodes where users actually login to the parallel machine.
From there, they can compile their code, and launch the job on “compute only” nodes — the worker
bees of the colony — and console information, machine status is communicated to the service nodes.
1.3 OVERVIEW OF PARALLEL PROGRAMMING MODELS
Early supercomputers were able to work efficiently, in part, because they shared a common physical
memory space. As a result, communication among processors was very efficient as they updated
shared variables and operated on common data. However, as the size of the systems grew, this
shared memory model evolved into a distributed shared memory (DSM) model where each processing
node owns a portion of the machines physical memory and the programmer is provided with a
logically shared address space making it easy to reason about how the application is partitioned and
communication among threads. The Stanford DASH [45] was the first to demonstrate this cache-
coherent non-uniform memory (ccNUMA) access model, and the SGI Origin2000 [43] was the
first machine to successfully commercialize the DSM architecture.
We commonly refer to distributed memory machines as“clusters” since they are loosely-coupled
and rely on message passing for communication among processing nodes. With the inception of
Beowulf clusters, the HPC community realized they could build modest-sized parallel computers on](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-24-320.jpg)
![1.4. PUTTING IT ALL TOGETHER 5
a relatively small budget. To their benefit, the common benchmark for measuring the performance
of a parallel computer is LINPACK, which is not communication intensive, so it was commonplace
to use inexpensive Ethernet networks to string together commodity nodes. As a result, Ethernet got
a foothold on the list of the TOP500 [62] civilian supercomputers with almost 50% of the TOP500
systems using Ethernet.
1.4 PUTTING IT ALL TOGETHER
The first Cray-1 [54] supercomputer had expected to ship one system per quarter in 1977. Today,
microprocessor companies have refined their CMOS processes and manufacturing making them
very cost-effective building blocks for large-scale parallel systems capable of 10s of petaflops. This
shift away from “proprietary” processors and trend toward “commodity” processors has fueled the
growth of systems. At the time of this writing, the largest computer on the TOP500 list [62] has in
excess of 220,000 cores (see Figure 7.5) and consumes almost seven megawatts!
A datacenter server has many commonalities as one used in a supercomputer, however, there
are also some very glaring differences. We enumerate several properties of both a warehouse-scale
computer (WSC) and a supercomputer (Cray XE6).
Datacenter server
• Sockets per server 2 sockets x86 platform
• Memory capacity 16 GB DRAM
• Disk capacity 5×1TB disk drive, and 1×160GB SSD (FLASH)
• Compute density 80 sockets per rack
• Network bandwidth per rack 1×48-port GigE switch with 40 down links, and 8 uplinks (5×
oversubscription)
• Network bandwidth per socket 100 Mb/s if 1 GigE rack switch, or 1 Gb/s if 10 GigE rack
switch
Supercomputer server
• Sockets per server 8 sockets x86 platform
• Memory capacity 32 or 64 GB DRAM
• Disk capacity IO capacity varies. Each XIO blade has four PCIe-Gen2 interfaces, for a total
of 96 PCIe-Gen2 ×16 IO devices for a peak IO bandwidth of 768 GB/s per direction.
• Compute density 192 sockets per rack](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-25-320.jpg)
![6 1. INTRODUCTION
• Networkbandwidthperrack 48×48-port Gemini switch chips each with 160 GB/s switching
bandwidth
• Network bandwidth per socket 9.6GB/s injection bandwidth with non-coherent Hyper-
Transport 3.0 (ncHT3)
Several things stand out as differences between a datacenter server and supercomputer node.
First, the compute density for the supercomputer is significantly better than a standard 40U rack. On
the other hand, this dense packaging also puts pressure on cooling requirements not to mention
power delivery. As power and its associated delivery become increasingly expensive, it becomes more
important to optimize the number of operations per watt; often the size of a system is limited by
power distribution and cooling infrastructure.
AnotherpointisthevastdifferenceinnetworkbandwidthpersocketinlargepartbecausencHT3
is a much higher bandwidth processor interface than PCIe-Gen2, however, as PCI-Gen3×16 be-
comes available we expect that gap to narrow.
1.5 QUALITY OF SERVICE (QOS) REQUIREMENTS
With HPC systems it is commonplace to dedicate the system for the duration of application ex-
ecution. Allowing all processors to be used for compute resources. As a result, there is no need
for performance isolation from competing applications. Quality of Service (QoS) provides both per-
formance isolation and differentiated service for applications2. Cloud computing often has a varied
workloads requiring multiple applications to share resources. Workload consolidation [33] is becom-
ing increasingly important as memory and processor cost increase, as a result so does the value of
increased system utilization.
The QoS class refers to the end-to-end class of service as observed by the application. In
principle, QoS is divided into three categories:
Best effort - traffic is treated as a FIFO with no differentiation provided.
Differentiated service - also referred to as “soft QoS” where traffic is given a statistical preference
over other traffic. This means it is less likely to be dropped relative to best effort traffic, for
example, resulting in lower average latency and increased average bandwidth.
Guaranteed service - also referred to as “hard QoS” where a fraction of the network bandwidth is
reserved to provide no-loss, low jitter bandwidth guarantees.
In practice,there are many intermediate pieces which are,in part,responsible for implementing a QoS
scheme. A routing algorithm determines the set of usable paths through the network between any
source and destination. Generally speaking, routing is a background process that attempts to load-
balance the physical links in the system taking into account any network faults and programming
2We use the term “applications” loosely here to represent processes or threads, at whatever granularity a service level agreement is
applied.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-26-320.jpg)
![1.6. FLOW CONTROL 7
the forwarding tables within each router. When a new packet arrives, the header is inspected and
the network address of the destination is used to index into the forwarding table which emits the
output port where the packet is scheduled for transmission.The “packet forwarding” process is done
on a packet-by-packet basis and is responsible for identifying packets marked for special treatment
according to its QoS class.
The basic unit over which a QoS class is applied is the flow. A flow is described as a tuple
(SourceIP, SourcePort, DestIP, DestPort). Packets are marked by the host or edge switch using
either 1) port range, or 2) host (sender/client-side) marking. Since we are talking about end-to-end
service levels, ideally the host which initiates the communication would request a specific level of
service. This requires some client-side API for establishing the QoS requirements prior to sending
a message. Alternatively, edge routers can mark packets as they are injected into the core fabric.
Packets are marked with their service class which is interpreted at each hop and acted upon by
routers along the path.For common Internet protocols,the differentiated service (DS) field of the IP
header provides this function as defined by the DiffServ [RFC2475] architecture for network layer
QoS. For compatibility reasons, this is the same field as the type of service (ToS) field [RFC791] of
the IP header. Since the RFC does not clearly describe how “low,” “medium,” or “high” are supposed
to be interpreted, it is common to use five classes: best effort (BE), AF1, AF2, AF3, AF4, and set
the drop priority to 0 (ignored).
1.6 FLOW CONTROL
Surprisingly, a key difference in system interconnects is flow control. How the switch and buffer
resources are managed is very different in Ethernet than what is typical in a supercomputer in-
terconnect. There are several kinds of flow control in a large distributed parallel computer. The
interconnection network is a shared resource among all the compute nodes, and network resources
must be carefully managed to avoid corrupting data, overflowing a buffer, etc.The basic mechanism
by which resources in the network are managed is flow control. Flow control provides a simple ac-
counting method for managing resources that are in demand by multiple uncoordinated sources.
The resource is managed in units of flits (flow control units). When a resource is requested but not
currently available for use, we must decide what to do with the incoming request. In general, we can
1) drop the request and all subsequent requests until the resource is freed, or 2) block and wait for
the request to free.
1.6.1 LOSSY FLOW CONTROL
With a lossy flow control [20, 48], the hardware can discard packets until there is room in the desired
resource. This approach is usually applied to input buffers on each switch chip, but also applies to
resources in the network interface controller (NIC) chip as well. When packets are dropped, the
software layers must detect the loss, usually through an unexpected sequence number indicating that
one or more packets are missing or out of order. The receiver software layers will discard packets
that do not match the expected sequence number, and the sender software layers will detect that it](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-27-320.jpg)
![8 1. INTRODUCTION
data
link
layer
data
link
layer
send
credits
data
packets
flow
ctrl
packets
Figure 1.4: Example of credit-based flow control across a network link.
has not received an acknowledgment packet and will cause a sender timeout which prompts the “send
window” — packets sent since the last acknowledgment was received — to be retransmitted. This
algorithm is referred to as go-back-N since the sender will “go back” and retransmit the last N (send
window) packets.
1.6.2 LOSSLESS FLOW CONTROL
Lossless flow control implies that packets are never dropped as a results of lack of buffer space (i.e.,
in the presence of congestion). Instead, it provides back pressure to indicate the absence of available
buffer space in the resource being managed.
1.6.2.1 Stop/Go (XON/XOFF) flow control
A common approach is XON/XOFF or stop/go flow control. In this approach, the receiver provides
simple handshaking to the sender indicating whether it is safe (XON) to transmit, or not (XOFF).
The sender is able to send flits until the receiver asserts stop (XOFF).Then, as the receiver continues
to process packets from the input buffer freeing space, and when a threshold is reached the receiver
will assert the XON again allowing the sender to again start sending. This Stop/Go functionality
correctly manages the resource and avoids overflow as long as the time at which XON is asserted
again (i.e., the threshold level in the input buffer) minus the time XOFF is asserted and the buffer
is sufficient to allow any in-flight flits to land. This slack in the buffer is necessary to act as a flow
control shock absorber for outstanding flits necessary to cover the propagation delay of the flow
control signals.
1.6.2.2 Credit-based flow control
Credit based flow control (Figure 1.4) provides more efficient use of the buffer resources.The sender
maintains a count of the number of available credits, which represent the amount of free space in
the receiver’s input buffer. A separate count is used for each virtual channel (VC) [21]. When a new](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-28-320.jpg)
![1.7. THE RISE OF ETHERNET 9
packet arrives at the output port, the sender checks the available credit counter. For wormhole flow
control [20] across the link, the sender’s available credit needs to only be one or more. For virtual
cut-through (VCT) [20, 22] flow control across the link, the sender’s available credit must be more
than the size of the packet. In practice, the switch hardware doesn’t have to track the size of the
packet in order to allow VCT flow control. The sender can simply check the available credit count
is larger than the maximum packet size.
1.7 THE RISE OF ETHERNET
It may be an extreme example comparing a typical datacenter server to a state-of-the-art super-
computer node, but the fact remains that Ethernet is gaining a significant foothold in the high-
performance computing space with nearly 50% of the systems on the TOP500 list [62] using Gi-
gabit Ethernet as shown in Figure 1.5(b). Infiniband (includes SDR, DDR and QDR) accounts
for 41% of the interconnects leaving very little room for proprietary networks. The landscape was
very different in 2002, as shown in Figure 1.5(a), where Myrinet accounted for about one third of
the system interconnects. The IBM SP2 interconnect accounted for about 18%, and the remaining
50% of the system interconnects were split among about nine different manufacturers. In 2002, only
about 8% of the TOP500 systems used gigabit Ethernet, compared to the nearly 50% in June of
2010.
1.8 SUMMARY
Nodoubt“cloudcomputing”benefitedfromthiswildgrowthandacceptanceintheHPCcommunity,
driving prices down and making more reliable parts. Moving forward we may see even further
consolidation as 40 Gig Ethernet converges with some of the Infiniband semantics with RDMA
over Ethernet (ROE). However, a warehouse-scale computer (WSC) [9] and a supercomputer have
different usage models. For example, most supercomputer applications expect to run on the machine
in a dedicated mode, not having to compete for compute, network, or IO resources with any other
applications.
Supercomputing applications will commonly checkpoint their dataset, since the MTBF of a
large system is usually measured in 10s of hours.Supercomputing applications also typically run with
a dedicated system, so QoS demands are not typically a concern. On the other hand, a datacenter
will run a wide variety of applications, some user-facing like Internet email, and others behind the
scenes. The workloads vary drastically, and programmers must learn that hardware can, and does,
fail and the application must be fault-aware and deal with it gracefully. Furthermore, clusters in the
datacenter are often shared across dozens of applications,so performance isolation and fault isolation
are key to scaling applications to large processor counts.
Choosing the “right” topology is important to the overall system performance. We must take
into account the flow control, QoS requirements, fault tolerance and resilience, as well as workloads
to better understand the latency and bandwidth characteristics of the entire system. For example,](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-29-320.jpg)
![13
C H A P T E R 2
Background
Over the past three decades, Moore’s Law has ushered in an era where transistors within a single
silicon package are abundant; a trend that system architects took advantage of to create a class of
many-core chip multiprocessors (CMPs) which interconnect many small processing cores using an
on-chip network. However, the pin density, or number of signal pins per unit of silicon area, has not
kept up with this pace. As a result pin bandwidth, the amount of data we can get on and off the chip
package, has become a first-order design constraint and precious resource for system designers.
2.1 INTERCONNECTION NETWORKS
The components of a computer system often have to communicate to exchange status information,
or data that is used for computation. The interconnection network is the substrate over which this
communication takes place. Many-core CMPs employ an on-chip network for low-latency, high-
bandwidth load/store operations between processing cores and memory,and among processing cores
within a chip package.
Processor, memory, and its associated IO devices are often packaged together and referred
to as a processing node. The system-level interconnection network connects all the processing nodes
according to the network topology. In the past, system components shared a bus over which address
and data were exchanged, however, this communication model did not scale as the number of
components sharing the bus increased. Modern interconnection networks take advantage of high-
speed signaling [28] with point-to-point serial links providing high-bandwidth connections between
processors and memory in multiprocessors [29, 32], connecting input/output (IO) devices [31, 51],
and as switching fabrics for routers.
2.2 TECHNOLOGY TRENDS
There are many considerations that go into building a large-scale cluster computer, many of which
revolve around its cost effectiveness, in both capital (procurement) cost and operating expense. Al-
though many of the components that go into a cluster each have different technology drivers which
blurs the line that defines the optimal solution for both performance and cost. This chapter takes a
look at a few of the technology drivers and how they pertain to the interconnection network.
The interconnection network is the substrate over which processors, memory and I/O devices
interoperate. The underlying technology from which the network is built determines the data rate,
resiliency,and cost of the network.Ideally,the processor,network,and I/O devices are all orchestrated](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-33-320.jpg)
![19
C H A P T E R 3
Topology Basics
The network topology — describing precisely how nodes are connected — plays a central role in
both the performance and cost of the network. In addition, the topology drives aspects of the switch
design (e.g., virtual channel requirements, routing function, etc), fault tolerance, and sensitivity to
adversarial traffic. There are subtle yet very practical design issues that only arise at scale; we try to
highlight those key points as they appear.
3.1 INTRODUCTION
Many scientific problems can be decomposed into a 3-D structure that represents the basic building
blocks of the underlying phenomenon being studied. Such problems often have nearest neighbor
communication patterns, for example, and lend themselves nicely to k-ary n-cube networks. A
high-performance application will often use the system dedicated to provide the necessary perfor-
mance isolation, however, a large production datacenter cluster will often run multiple applications
simultaneously with varying workloads and often unstructured communication patterns.
The choice of topology is largely driven by two factors: technology and packaging constraints.
Here, technology refers to the underlying silicon from which the routers are fabricated (i.e., node size,
pin density, power, etc) and the signaling technology (e.g., optical versus electrical). The packaging
constraints will determine the compute density, or amount of computation per unit of area on the
datacenter floor. The packaging constraints will also dictate the data rate (signaling speed) and
distance over which we can reliably communicate.
As a result of evolving technology,the topologies used in large-scale systems have also changed.
Many of the earliest interconnection networks were designed using topologies such as butterflies or
hypercubes, based on the simple observation that these topologies minimized hop count. Analysis
by both Dally [18] and Agarwal [5] showed that under fixed packaging constraints, a low-radix
network offered lower packet latency and thus better performance. Since the mid-1990s, k-ary
n-cube networks were used by several high-performance multiprocessors such as the SGI Origin
2000 hypercube [43], the 2-D torus of the Cray X1 [16], the 3-D torus of the Cray T3E [55]
and XT3 [12, 17] and the torus of the Alpha 21364 [49] and IBM BlueGene [35]. However, the
increasing pin bandwidth has recently motivated the migration towards high-radix topologies such
as the radix-64 folded-Clos topology used in the Cray BlackWidow system [56]. In this chapter, we
will discuss mesh/torus topologies while in the next chapter, we will present high-radix topologies.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-39-320.jpg)
![20 3. TOPOLOGY BASICS
3.2 TYPES OF NETWORKS
Topologies can be broken down into two different genres: direct and indirect [20]. A direct network
has processing nodes attached directly to the switching fabric; that is, the switching fabric is dis-
tributed among the processing nodes. An indirect network has the endpoint network independent
of the endpoints themselves – i.e., dedicated switch nodes exist and packets are forwarded indirectly
through these switch nodes. The type of network determines some of the packaging and cabling
requirements as well as fault resilience. It also impacts cost, for example, since a direct network can
combine the switching fabric and the network interface controller (NIC) functionality in the same
silicon package. An indirect network typically has two separate chips, with one for the NIC and
another for the switching fabric of the network. Examples of direct network include mesh, torus, and
hypercubes discussed in this chapter as well as high-radix topologies such as the flattened butterfly
described in the next chapter. Indirect networks include conventional butterfly topology and fat-tree
topologies.
The term radix and dimension are often used to describe both types of networks but have been
used differently for each network. For an indirect network, radix often refers to the number of ports
of a switch, and the dimension is related to the number of stages in the network. However, for a
direct network, the two terminologies are reversed – radix refers to the number of nodes within a
dimension, and the network size can be further increased by adding multiple dimensions. The two
terms are actually a duality of each other for the different networks – for example, in order to reduce
the network diameter, the radix of an indirect network or the dimension of a direct network can be
increased. To be consistent with existing literature, we will use the term radix to refer to different
aspects of a direct and an indirect network.
3.3 MESH,TORUS, AND HYPERCUBES
The mesh,torus and hypercube networks all belong to the same family of direct networks often referred
to as k-ary n-mesh or k-ary n-cube.The scalability of the network is largely determined by the radix,
k,and number of dimensions,n,with N = kn total endpoints in the network.In practice,the radix of
the network is not necessarily the same for every dimension (Figure 3.2). Therefore, a more general
way to express the total number of endpoints is given by Equation 3.1.
N =
n−1
i=0
ki (3.1)
4
3
2
1 6
5 7
0 4
3
2
1 6
5 7
0
(a) 8-ary 1-mesh. (b) 8-ary 1-cube.
Figure 3.1: Mesh (a) and torus (b) networks.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-40-320.jpg)
![25
C H A P T E R 4
High-Radix Topologies
Dally [18] and Agarwal [5] showed that under fixed packaging constraints, lower radix networks
offered lower packet latency. As a result, many studies have focused on low-radix topologies such as
the k-ary n-cube topology discussed in Chapter 3.The fundamental result of these authors still holds
– technology and packaging constraints should drive topology design. However, what has changed
in recent years are the topologies that these constraints lead us toward. In this section, we describe
the high-radix topologies that can better exploit today’s technology.
(a) radix-16 one-dimensional torus with each unidirectional link L lanes wide.
(b) radix-4 two-dimensional torus with each unidirectional link L/2 lanes wide.
Figure 4.1: Each router node has the same amount of pin bandwidth but differ in the number of ports.
4.1 TOWARDS HIGH-RADIX TOPOLOGIES
Technology trends and packaging constraints can and do have a major impact on the chosen topology.
For example, consider the diagram of two 16-node networks in Figure 4.1. The radix-16 one-
dimensional torus in Figure 4.1a has two ports on each router node; each port consists of an input](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-45-320.jpg)
![4.2. TECHNOLOGY DRIVERS 27
where L is the length of a packet, and b is the bandwidth of the channels. For an N node network
with radix k routers (k input channels and k output channels per router), the number of hops1 must
be at least 2logkN. Also, if the total bandwidth of a router is B, that bandwidth is divided among
the 2k input and output channels and b = B/2k. Substituting this into the expression for latency
from Equation (4.1)
T = 2tr logk N + 2kL/B (4.2)
Then, setting dT/dk equal to zero and isolating k gives the optimal radix in terms of the network
parameters,
k log2
k =
Btr log N
L
(4.3)
In this differentiation, we assume B and tr are independent of the radix k. Since we are evaluating
the optimal radix for a given bandwidth, we can assume B is independent of k. The tr parameter is
a function of k but has only a small impact on the total latency and has no impact on the optimal
radix. Router delay tr can be expressed as the number of pipeline stages (P) times the cycle time
(tcy). As radix increases, the router microarchitecture can be designed where tcy remains constant
and P increases logarithmically. The number of pipeline stages P can be further broken down into
a component that is independent of the radix X and a component which is dependent on the radix
Y log2 k. 2 Thus, router delay (tr) can be rewritten as
tr = tcyP = tcy(X + Y log2 k) (4.4)
If this relationship is substituted back into Equation (4.2) and differentiated, the dependency on
radix k coming from the router delay disappears and does not change the optimal radix. Intuitively,
although a single router delay increases with a log(k) dependence, the effect is offset in the network
by the fact that the hop count decreases as 1/ log(k) and as a result, the router delay does not
significantly affect the optimal radix.
In Equation (4.2), we also ignore time of flight for packets to traverse the wires that make
up the network channels. The time of flight does not depend on the radix(k) and thus has minimal
impact on the optimal radix. Time of flight is D/v where D is the total physical distance traveled
by a packet, and v is the propagation velocity. As radix increases, the distance between two router
nodes increases. However, the total distance traveled by a packet will be approximately equal since
the lower-radix network requires more hops. 3
From Equation (4.3),we refer to the quantity A = Btr log N
L as the aspect ratio of the router [42].
This aspect ratio impacts the router radix that minimizes network latency.A high aspect ratio implies
a “tall, skinny” router (many, narrow channels) minimizes latency, while a low ratio implies a “short,
fat” router (few, wide channels).
1Uniform traffic is assumed and 2logkN hops are required for a non-blocking network.
2For example, routing pipeline stage is often independent of the radix while the switch allocation is dependent on the radix.
3The time of flight is also dependent on the packaging of the system but we ignore packaging in this analysis.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-47-320.jpg)
![28 4. HIGH-RADIX TOPOLOGIES
1996
2003
2010
1991
1
10
100
1000
10 100 1000 10000
Aspect Ratio
Optimal
Radix
(k)
Figure 4.2: Relationship between the optimal radix for minimum latency and router aspect ratio. The
labeled points show the approximate aspect ratio for a given year’s technology with a packet size of L=128
bits
0
50
100
150
200
250
300
0 50 100 150 200 250
radix
latency
(nsec)
2003 technology 2010 technology
0
1
2
3
4
5
6
7
8
0 50 100 150 200 250
radix
cost
(
#
of
1000
channels)
2003 technology 2010 technology
(a) (b)
Figure 4.3: Latency (a) and cost (b) of the network as the radix is increased for two different technologies.
A plot of the minimum latency radix versus aspect ratio is shown in Figure 4.2 annotated with
aspect ratios from several years.These particular numbers are representative of large supercomputers
with single-word network accesses4, but the general trend of the radix increasing significantly over
time remains. Figure 4.3(a) shows how latency varies with radix for 2003 and 2010 aspect ratios. As
radix is increased, latency first decreases as hop count, and hence Th, is reduced. However, beyond a
certain radix, serialization latency begins to dominate the overall latency and latency increases. As
bandwidth, and hence aspect ratio, is increased, the radix that gives minimum latency also increases.
For 2004 technology (aspect ratio = 652), the optimum radix is 45 while for 2010 technology (aspect
ratio = 3013) the optimum radix is 128.
4The 1996 data is from the Cray T3E [55] (B=48Gb/s, tr =40ns, N=2048), the 2003 data is combined from the Alpha 21364 [49]
and Velio VC2002 [20] (1Tb/s, 10ns, 4096), and the 2010 data was estimated as (20Tb/s, 2ns, 8192).](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-48-320.jpg)
![comparison with the growth of a theory will always throw light on
the true nature of the will and its continuous effort to satisfy itself.
2. Right or Law may be taken in the widest sense as including the
whole manifestation of Will in an actual world—“the actual body of
all the conditions of freedom,” [1] “the realm of realised freedom,
the world of mind produced out of itself, as a second nature.” [2] It
is a merit of the German {259} term “Recht” [3] that it maintains
the connection between the law and the spirit of law, [4] and almost
of itself prohibits the separation between positive law, and will,
custom or sentiment, which underlies such a theory as Austin’s.
[1] Hegel, Philos. of Mind (E. Tr.), p. 104. Cf. defs. quoted from
Green, p. 203 above.
[2] Rechtsphil., sect. 4.
[3] Cf. the Greek’s idea of “nomos.”
[4] See ch. ii. above on Montesquieu and Rousseau.
This whole sphere of Right or Law, the mind as actualised in
Society and the State, naturally divides itself on the principle which
has just been explained, into three connected groups of ideal facts
or points of view. The first, or simplest and most inevitable, of these,
may be called the “letter of the law” as we come upon it most
especially in the law of property—Shylock’s law—the sheer fact, as it
seems, that the world is appropriated by legal “persons.”
The second, obviously conditioned by the first both positively and
negatively, may be described as the morality of conscience; the
revolt of the will against the letter of the law, though this was its](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-53-320.jpg)
![And thirdly, there is the reality or concrete experience in which the
two former sets of facts, or ideas, find their true place and
justification—the completed theory, so to speak, which adjusts and
explains the narrower views founded on one-sided contact with life.
This is indicated to consist in “social observance,” or “ethical use and
wont”; the system of working mind where the true will appears as
incarnate in a way of living. This, {260} like the others, it must be
remembered, is a fact, though akin to a theory. Not only does it
explain and justify the other factors, but its existence has enabled
them to exist, as theirs has also been essential to it. And yet each of
the three, as one aspect of society which under certain influences
may catch the eye, has at times claimed—is, indeed, constantly
claiming—predominance, and has thus brought into relief its own
defects and the need of the complementary ideas. We will speak of
these moods of mind or kinds of experience in their order, expecting
a further sub-division when we come to treat of the third.
3. “Law,” then, in the directest possible sense—the minimum
sense, so to speak—is the hard literal fact that it is a rule of the
world we live in for things to be appropriated by persons. This is the
first or minimum change of the world from mere matter into the
instruments of mind, and it is a necessary change. Things have no
will of their own, and it is by having a will asserted upon them that
they become organs of life. In the same way, it is by assertion in
external things that the will first becomes a fact in the material
world. Property is “the first reality of freedom.” [1] It is not the mere
provision for wants, but the material counterpart of will. Contract
belongs to this sphere, the sphere of property. It is an agreement of
persons about an external thing—a “common will,” but not one](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-55-320.jpg)
![“general” or “universal” in its own nature like that involved in the
State.
[1] Rechtsphil., sect. 41. Not, in its developed form, the first in
time. Hegel lays stress on the fact that true, free, property was
hardly realised even in his own day.
{261} Thus, it is a confusion of spheres to apply the idea of
contract to the State, for the State is an imperative necessity of
man’s nature as rational, while contract is a mere agreement of
certain free persons about certain external things. The idea of the
social contract is a confusion of the same type as that by which
public rights and functions were treated as private property in the
middle age. The attributes of private property are nothing more than
the conditions of “personal” existence, and absurdity results if they
are transferred to functions of the State.
This phase or view of law as, in its letter, an ultimate and absolute
rule, may be illustrated, Hegel says, by the Stoic notion that there is
only one virtue and one vice; by the Draconic conception that every
offence demands the extreme penalty; and by “the barbarity of the
formal code of honour, which found in every injury an unpardonable
insult.” It might also be illustrated by Austin’s theory of law as a
command enforced by a penalty; or by the theories which account
for property simply by the fact of occupancy or of labour mixed with
the thing. The common point of all these views is that they treat the
law, not as a part of a living system, [1] ultimately resting on the will
to maintain a certain type of life, but as something absolute in its
separateness, and equally sacred in all its accidents and inequalities.
[1] See e.g. above, p. 232, how the idea of a system of rights may
modify punishment.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-56-320.jpg)
![Now, this emphasis and idea of law, being the exaggeration of a
single and direct necessity, the {262} necessity of order and
property, may be called “primitive” or barbarous, but it cannot of
course be identified with the earliest state of social authority known
to history or to anthropology. There we should probably find law
undifferentiated from custom and from religious sentiment, and
consequently, though rigid enough, not in any such one-sided
absoluteness as we have been describing. All we can say is that this
is the way in which law must come to be regarded whenever its
living spirit is forgotten, and an unreal absoluteness is assigned to it;
and this connection of principle verifies itself as a fact in recurrent
historical phenomena, and in fallacies which perpetually reappear.
4. Within the whole fabric of right or realised will, the element
which naturally asserts itself by antagonism to the letter of the law is
the morality of conscience, conscientiousness, or the idea of the
Good Will. It is connected with the letter of the law, as Hegel puts it,
by the various degrees of wrong. The will, that is to say, finds itself
at variance in or with [1] the order of law and property which it has
created as its direct and necessary step to freedom. Its realised
theory, so to speak, is found to break down at a certain point, by
being in contradiction with the needs which it was created to meet.
“Summum jus, summa injuria”. We may object that the anti-legal will
is simply wrong. This may be so, and again it may not be so. What
the will has awakened to, whether right or wrong, is {263} that it
can acquiesce in nothing which does not come home to it as fulfilling
its own principle. What so comes home to it is what it calls “good,”
and it cannot accept any order or necessity which it cannot will as
good, i.e. as satisfying its own idea.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-57-320.jpg)
![[1] “In it,” when my will does not conflict with right as such, but
claims the right in an object A to be mine and not yours—a civil
dispute. “With it,” when my will rebels, and by its act, so far as in it
lies, denies and destroys the whole fabric of right, e.g. takes the
object A, without alleging a right to it—theft, a criminal offence, cf.
p. 230.
When this phase of reaction is pushed to its logical extreme, we
have the modern doctrine of my conscience and my pure will. It is
the conflict of the inner self with the outer world, expressed in
history through the Stoic and through some forms of the Christian
consciousness (especially the Protestant consciousness), and in
philosophy through the Kantian doctrine of the good will, uttered in
the famous sentence, “Nothing can possibly be conceived in the
world or out of it which can be called good without qualification
except a good will.” [1] Nothing is worth doing but what one ought,
and because one ought.
[1] Kant, Grundlegung zur Metaphysik d. Sitten, sect. I.
The criticism to which this principle has been subjected is familiar
to students of ethics. Its point is, in brief, that there is no way of
connecting any particular action with the mere idea of a pure will.
The forms assumed by evasions of this difficulty, which we fall into
when we desire wholly to separate the inner from the outer, or the
“ought” from the “is,” are treated by Hegel with unsurpassable
vigour and subtlety, as indeed the annihilating criticism of this
conception is primarily due to him. The essence of the matter is that
the pure will directed towards good for the sake of good, having no
real connection with any detailed conduct, may be alleged by self-
deception in support of any behaviour whatever, and out of this may](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-58-320.jpg)
![spring the {264} whole sophistry and hypocrisy of “pure intention.”
He makes the shrewd observation, [1] which is still of interest, that
the extreme Protestant doctrine of conscience may take the form of
ethical vacuity or instability, and that this had in his time been the
cause of many Protestants going over to Rome, to secure some sort
of moorings, if not precisely the stability of thought.
[1] Rechtsphil., sect. 141.
Still, out of all this one-sidedness, there survives the permanent
necessity that an intelligent being can acquiesce only in what enters
into the object of his will. It is his will which affirms the aim to which
his nature draws him, and he is absolutely debarred from reposing in
anything which does not appeal to his will. The subjective will is the
only soil on which freedom can be a reality.
So, within the general organism of Right or realised Free-will, we
have found two opposite groups of facts—for the aspirations of
intelligent beings are facts—or tendencies or theories, which are
connected by opposition, and yet are necessary to the expression of
the same underlying need—the letter of the law, and the freedom of
conscience.
5. Hegel’s name for the third term, which, as he puts it, expresses
the “truth” of these extremes, may be rendered “the Ethical System,”
or “the Moral Life,” or “Social Ethics.” It expresses “the truth” of the
extremes, as a good theory may express the truth of two one-sided
views. Only, as we have said, it is a fact as well as a theory, and
therefore is something which actually contains what these two views
demand, and does the work which they, and the facts they rely on,
{265} exhibit as necessary to be done. This relation is not obscure
or unprecedented. Every institution, every life, works as a theory,](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-59-320.jpg)
![and either masters its facts or fails to master them; though not
every theory is a life or an institution.
The German word which the above-mentioned phrases attempt to
render is “Sittlichkeit” The word takes its meaning from “Sitte” which
in common usage is equivalent to “custom.” Hegel’s use of the term,
in his later writings, as opposed to “Moralität” and as indicating, in
comparison with it, a fuller and truer phase of life, is an intentional
declaration of war against the Kantian principle of the pure good will,
and is the gist of Hegel’s ethico-political view in a nutshell. The word
would most naturally apply to the life of a community in which law,
custom, and sentiment were not yet very sharply distinguished.
According to accepted views, the communities of ancient Greece,
before they were stirred by the reflective movement which is
associated with the names of Socrates and the Sophists, would be
examples of a disposition and order of life which the word
“Sittlichkeit” might denote. And it was in the Greek communities, as
is shown by the work which he sketched as early as 1802, [1] that
Hegel found this suggestion of a whole in which law and custom,
duty and disposition, were absolutely at one. He subsequently
modified the conception in accordance with the modern idea of
freedom, by allowing a greater emphasis and relief to its {266}
component parts, and insisting (against Plato’s Republic for
instance,) on the principle of individual choice, initiative, and
property, as necessary to the complete communion of intelligent
beings. As we have just seen, indeed, he introduces reflective
morality or conscientiousness into the sphere of Right, to represent
the full nature of mind, which is only exhibited in a consciousness
which pursues its aims of its own choice and for their own sake.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-60-320.jpg)
![[1] The System d. Sittlichkeit. The Rechtsphil. was not published
till 1817, in its earliest form. See Wallace, Hegel’s Philosophy of
Mind, p. 187.
The Ethical System, then, or Social Ethics, is put forward as the
ideal fact which includes, and does the work of both the literal law
and the moral will, alike in practice and as a theory. It is the idea of
freedom developed (i.) into a present world, and (ii.) into the nature
of self-consciousness.
For (i.), in the first place, the ethical system, or the ways of acting
which make up social ethics, constitute a present and actual world.
So far it partakes of the nature of the literal law and order, the
system of property-holding, which, as we have seen, is all but a
natural fact. Social Ethic, we might say, is a physical fact. The bodily
habits and external actions of a people incorporate it. It transforms
the face of a country, “domesticating the untamed earth.” [1] Each
individual has his own bodily existence in a determinate mode as a
part of the ethical life of society. The rules and traditions of ethical
living are, as has been said, “the nature of things.” They are as hard,
as “objective” an order as “sun, moon, mountains, rivers, and all
objects of nature.” [2] Man lives according to them before he knows
that he {267} does so, and always, in a great degree, independently
of knowing that he does so. As this group of facts, or considered
from this point of view, the ethical system is the body of the moral
world.
[1] Aeschylus, Eumenides, I. 14. [2] Rechtsphil., sect. 146.
(ii.) But it is also and no less the very nature of self-consciousness.
It is as much a demand of man’s intelligence or an inner and
universal law as the “pure will” itself. [1] The difference between](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-61-320.jpg)
![them is that the Ethical System is a system, a world, though from
the point of view of will regarded as inner, that is to say, as
something which is the motive and fulfills the demand of
consciousness. Thus, it bears the character of a thoroughly
systematised theory, as contrasted with the idea of a good will,
which is a mere general point of view. And it is because of this
systematic character that it is enabled to connect the individual or
particular will with the universal spirit of the community. It is only in
a system that a particular fact can be connected with a universal
law, as the planetary motions are with the law of gravitation. The
particular will, as we have explained above, is universalised by its
relation to a systematic purpose which it partly implies and partly
realises. A man wishes for this thing or that thing, but not at any
price. The reservations to which his wish is subject, by reason of
other purposes and postulates of life, are known to him only in part;
but if they could be stated in full, they would constitute the system
of his life as realised in the universal life of the community. It is
precisely {268} analogous to the process which a common judgment
of perception has to go through in becoming a scientific truth—the
implications have to be stated in full, and the perception modified in
accordance with them. And when this is done, we have no longer a
fact, but a science.
[1] On all this portion of the subject, see Mr. Bradley’s Essay, “My
Station and its Duties,” in Ethical Studies.
Regarded from this point of view, as the substance of the
individual
Will, the Ethical System is the Soul of the moral world.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-62-320.jpg)
![All those theories, therefore, which tend to assimilate the State to
a family by a sort of levelling down of the former or levelling up of
the latter (Plato’s Republic, the phalanstery, paternal government,
and the like) involve for Hegel a mere confusion of relations. They
recognise an element which is essential to society, and may truly be
said to be even its foundation. But they do not see its right place in
the whole, and do not understand that in order to attain a stronger
and deeper unity (which is, in short, a stronger and deeper mind)
the different elements must be allowed a greater emphasis and
relief, and their respective characteristics must not be slurred or
scamped.
But (b) in the second place, the family is a factor in the rational
whole, the State, though its function par excellence is that of the
natural basis of society. Hence its nature and sanction is ethical—it
rests neither on mere feeling on {271} the one hand, nor on mere
contract on the other. It has a public side, and the acceptance of a
universal obligation by a declaration in explicit language (language
being the stamp of the universal), in face of the community, is an
essential part of marriage, and not a mere accident or accessory, as
the votaries of feeling have urged. This view is aimed against the
confusion which finds the sole essence of marriage in feeling. This is
a perpetually recurring contention, represented in Hegel’s day by
Friedrich von Schlegel’s Lucinde, which argues that the form of
marriage destroys the value of passion. Hegel’s analyses are
everywhere directed against this inability to grasp the distinct sides
of a many-sided fact.
(c) The ethical aspect of the family [1] shows itself in the nature
and organisation of the household, as an institution embodying
permanent interests and relations of the two persons who are its](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-65-320.jpg)
![head, and as an organ of public duties in the bodily and spiritual
nurture of the children. The permanent and equal relation of the
heads of the household, involved in its nature as the ethical aspect
of the family, implies monogamy, and it is the monogamous family
alone which can count as a true element of the ethical order.
[1] Cf. Green’s Principles of Political Obligation, p. 235.
(d) The household, being the true and operative ethical organ
which makes parentage into family, is the unit which demands to be
respected and protected by the State against the less differentiated
forms of consanguinity, such as the clan. The true family starts from
marriage and the foundation of a household, and in the early {272}
development of law we find the State, with a just instinct, protecting
the household against the clan, e.g. by conferring the power of
bequest. This power, though now it may imply a discretion mainly
hostile to the family, presented itself in early law rather as a means
of perpetuating the separate household as against the pretensions of
the clan to interfere with its property.
Thus, the monogamous family is naturally and necessarily, to
some extent, a unit in respect of property; the children, at least,
being inevitably under tutelage and incapable of self-support, even if
economic equality asserts itself as between husband and wife. This
peculiar relation in respect of property is rooted in the unique nature
of the household, as an organ for the guardianship of immature
lives, and as a unity of feeling rather than of explicit thought. It is
noticeable that progress tends to introduce the distinctions of
property within its unity [1] (though for children this can never go
very far), and very slightly to introduce the relations of the family
into the outside world. In as far as such distinctions come to be](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-66-320.jpg)
![made, the nature and functions of the household being undisturbed,
a somewhat higher intensity of ethical union is rendered necessary,
and will no doubt assert itself.
[1] Married Women’s Property, Protection of Earnings of Children,
Property assigned by understanding within household to young
children.
7. When the man (or woman [1]) arrives at maturity and leaves
the safe harbour of the family, he finds himself, prima facie, isolated
in a world {273} of conflicting self-interests. He has his living to
make, or his property to administer. He is tied to others, in
appearance, only by the system of wants and work, with the
elementary function which is necessary to it, viz. its police functions
and the administration of justice.
[1] Hegel would say only or chiefly the man, who is for him the
natural earner and chief of the household.
It is this phase of social life, and the temper or disposition
corresponding to it, which Hegel indicates by the expression
Bourgeois Society. [1] It presents itself to him as the opposite
extreme of life and mind to that embodied in the family. It is an
aggregate of families—for the units of the Bourgeois Society are
heads of households—as seen from the outside, in the great system
of industry and business, where a man has to find his work and do
it. It is, in mind, the presence of definite though limited aims,
calculation and self-interest. [2]
[1] Bürgerliche Gesellschaft. “Society,” Wallace points out, is here
opposed to “community,” and indicates a looser phase of union.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-67-320.jpg)
![[2] Cf. the merchant in Wilhelm Meister’s Lehrjahre, viii. 2. “I can
assure you that I never reflected on the State in my life. My tolls,
charges, and dues I have paid for no other reason than that it was
established usage” (cited from Wallace, Hegel’s Philosophy of Mind,
p. cci.).
Bourgeois Society is the aspect of the social whole insisted on by
the classical political economy, by which, as an achievement in the
way of reducing complex appearances to principles, Hegel was much
impressed. It is, again, the view of society embodied in the
conception of the purely police State, and its principle is confused
with that of the State proper by one set of theorists, as that of the
family is by another.
It is the peculiarity of Hegel’s view—probably {274} the most
definitely original, as it is the most famous, of all his political ideas—
to contend that this aspect of society, with the form of consciousness
belonging to it, is necessary to a modern State. According to his
logic, indeed, it is inevitable that every true whole shall have an
aspect of “difference,” of breaking up into particulars.
The principle of the ancient State, as concentratedly expressed in
Plato’s Republic, was weak and undeveloped, and fell short of the
true claims of intelligence, [1] just because it dared not really let the
individual go—let him assert himself as himself. “Subjectivity” was a
principle fatal to it. Not that there was an iron oppression in the
States of antiquity. The individual was, for an onlooker, magnificently
developed. His limitations were in him, and did not oppress him; but
for all that, free choice and the career open to talents were not for
him.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-68-320.jpg)
![[1] “Was not ideal enough” (Hegel, Geschichte der Philosophie, ii.
254). The “notion” for him necessarily involves identity,
differentiation, and re-integration; and in this respect the ancient
State falls short of a true notion, while the modern realises it.
The modern demand—such is Hegel’s conception—is harder and
higher. The individual’s life is not predetermined by his birth, but he
is thrown face to face with economic necessity, which is a form of
the universal end. He has to strip off his crudeness and vanity, and,
of himself, mould himself into something which fulfils a want. This is
a step without which there can be no true freedom—the giving one’s
self by one’s own act a definite place in the region of external
necessity, the “becoming something” or attaching oneself to {275} a
definite class of service renderers. Thus, we are startled to find
culture or education treated in general, and in respect of its
indispensableness, under the head of the Bourgeois Society. For
culture is the liberation from one’s own caprices, and the acceptance
of a universal task. It is a severe process, and therefore unpopular,
but it is a necessary one if we are to have true freedom. The
criticism that such a world and temper is the world and temper of
self-interest does not appeal strongly to Hegel. We shall have to
treat of it more fully below. [1]
[1] See p. 291.
It may be noted in passing that the insecurity of life, which may
seem to attach to dependence on the vast system of wants and
work, is more and more seen, as modern economic relations
develop, not to be insecurity at all, except in as far as “culture” in
the form of industrial training is absent. There is, indeed, in modern
life, nowhere any absolute and oyster-like stability. The highest](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-69-320.jpg)
![stability to be anywhere attained is that due to fitness for service in
the interdependent system of needs. [1]
[1] I may refer to The Standard of Life, by H. Bosanquet, essay on
“Klassenkampf”.
Therefore, as Hegel saw, but in more ways than he saw, the
system of Bourgeois Society—the economic and industrial world—is
not a separate reality, but only an appearance within a larger
system. The member of it is not so detached as he may seem, or
think. He is within, and sustained by, the general life of the State, as
the aims which are his motives in “business” or industry are within
{276} and inseparable from the whole structure of his intelligence.
Thus, the world of Bourgeois Society—a world, on the whole, of
cash nexus and mere protection by the State—has a structure or
tendency of its own which brings it back by necessary steps to
connection with the State proper or explicit and determinate social
unity. It is, we must observe, posterior to the State in time. It is only
within the State proper, and resting on its solid power, that such a
world as that of Bourgeois Society could arise or be conceivable. Its
priority to the State is, like that of the family, the priority of
comparative narrowness or simplicity, of dealing with fewer factors,
and of representing human nature in a more special, though
necessary, aspect. And for this very reason it could not exist by itself.
It has not the many-sided vitality indispensable to anything which is
to hold its own in the actual world.
The working of the Bourgeois Society, then, exhibits an inevitable
connection with the State proper, and, so to speak, leads up to it.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-70-320.jpg)
![In the first place, the economic world implies the administration of
justice. In this, as involving a developed system of civilised law,
there is an advance on the “letter of the law” in its crudest and most
barbarous acceptation. The system of law of a modern State is, and
still more ought to be, [1] a fairly reasonable and intelligible
definition of the rights and relations of persons. By this
determination the economic system of particular wants and services
enters upon a first {277} approximation, as it were, to a unity of
principle. The law only professes, indeed, to protect property and
exchange, but in doing so it unavoidably recognises that the
particular want has a general bearing; for the developed system of
law only comes into existence to enable wants to be supplied, and
takes its definite shape according to the system of wants. We may
illustrate this first approximation to universality, which law confers
upon the particulars of private interest, by a suggestive view which
M. Durkheim has propounded. [2] He has pointed out that the
current formula for social change, “from status to contract,” has a
subtler significance than is apt to be recognised. For contract is not
really indeterminate, as if it arose in vacuo without a precedent. It
runs in forms determined by social experience through law and
custom; and thus the law, which professedly aims at protecting
property and exchange, necessarily regulates them by the modes in
which it chooses to protect them.
[1] Hegel pleads strongly for codification.
[2] De la Division du Travail Social, 225 ff.
A more intimate relation to the State proper—to a definite
principle, as we might say, of common good—grows out of the
interests of Bourgeois Society which take the shape of what a](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-71-320.jpg)
![German calls “Police and Corporation,” i.e. State regulation and
Trade Societies.
The basis of State regulation is the emergence of aspects of
common interest in the system of particular interests. The region of
particular interests (supply and demand) has an accidental side, and
the State has a right and a duty to protect the general good against
accidental {278} hindrances. On the whole, no doubt, the right
relation between producer and consumer arises of itself, but
miscarriages may occur which call for interference on behalf of the
explicit [1] principle of the general good. The general possibility of
the individual’s obtaining what he wants is a public interest, and the
State has a right to intervene with this end in view, both by
execution of necessary public works, by sanitary inspection and the
like, and by inspection and control of fraud in the case of necessary
commodities offered for sale to the general public. For the public
offer of goods in daily use is not a purely private concern, but a
matter of the general interest. If indeed there was complete official
regulation, there would be a risk of getting work like the Pyramids,
that represented no private want at all; but yet, in the system of
private wants, there is a public interest that demands vigilance.
[1] The explicit idea of common good always belongs in Hegel to
the State proper.
A similar approximation of Bourgeois Society to the State is
constituted by the “Corporation,” which rests on the facts of class.
Every member of the Bourgeois Society belongs by his vocation to a
class, and this breaking up into classes is a consequence of the
division of labour which prevails in the economic sphere, disguising
the common good as private interest or necessity. But in the](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-72-320.jpg)
![formation of classes society begins as it were to recover from the
dispersion which private interest has occasioned. As a member of his
class [1] or {279} “estate,” the citizen acquires solidarity with his
fellows, and his particular interest becomes ipso facto a common
one. As a member of the class, again, he is, or ought to be, a
member of his “trade society” or “corporation.” In this he finds his
honour or recognition, [2] a definite standard of life (apart from
which he is apt to assert himself by aimless extravagance, for want
of a recognised respectability), a standard of work, insurance against
misfortune, and (as a candidate for admission) the means of
technical education.
[1] The term “Stände” it must be remembered, has for a German
the association of elements of the representative assembly; “états”,
estates of the realm.
[2] Cf. the English workman’s phrase, “a good tradesman,” i.e. a
competent member of his trade.
If the family is the first basis of the State, the classes or estates
are the second. The Corporation or Trade Society is a second family
to its members. It is the very root of ethical connection between the
private and the general [1] interest, and the State should see to it
that this root holds as strongly as possible. [2]
[1] “We can only say that these men, if they leave us, will bitterly
regret it. … The man who is so unselfish as to care nothing for
himself or his fellow-men will soon find himself, as years creep over
him, and grey hairs and glasses, completely cut out.”—“Branch Trade
Report (Birmingham) to National Union of Boot and Shoe Operatives,
January, 1896.”](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-73-320.jpg)
![[2] Sects. 201 and 255. I omit Hegel’s characterisation of the
classes, which has a good deal in common with theories which
represent occupations as determining character. The contrast
between agricultural and industrial or commercial life, between
country and town, is of great importance in his view. He almost
seems to confine Bourgeois industrialism as such to the life of town-
dwellers; though, again, ultimately the whole division into classes is
characteristic of Bourgeois Society (cf. sects. 256 and 305).
“If,” Hegel writes, [1] “in recent days the “Corporation has been
abolished, this has the significance {280} that the individual
ought to provide for himself. This may be admitted; but the
corporation did not alter the individual’s obligation to earn his
livelihood. In our modern States the citizens have only a limited
share in the universal business of the State; but it is necessary to
permit the ethical human being a universal activity over and
above his private end. This universal, which the modern State
does not always provide for him, he finds in the Corporation. We
saw before that the individual providing for himself in the
Bourgeois Society also acts for others. But this unconscious
necessity is not enough; it needs the Corporation to bring it to a
conscious and thoughtful social ethics. Of course the Corporation
needs the higher superintendence of the State, or it would ossify,
shrink into its shell, and be degraded into a wretched guild. But in
and for itself the Corporation is no closed guild; it is rather the
bringing of an isolated trade into an ethical connection, and its
admission into a sphere in which it wins strength and honour.” [2]
[1] Sect. 255.](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-74-320.jpg)
![[2] It is obvious that this treatment of associations arising among
classes in industry and commerce does not apply in principle
exclusively to trade or professional societies. It would include, e.g.,
Friendly Societies and Co-operative Societies, by which members of
the economic world bind themselves together for help, recognition,
and the assertion of their general interests.
8. The State proper, or political constitution, presents itself to
Hegel as the system in which the family and the Bourgeois Society
find their completion and their security. He was early impressed, as
we have seen, with the beautiful unity of the ancient Greek
commonwealths. And the first and last idea which governs his
representations of the modern State is that of the Greek
commonwealth enlarged as it was from a sun to a solar {281}
system. The family feeling and the individual interest are in the
modern State let go, accented, intensified to their uttermost power;
and it is out of and because of this immense orbit of its elements
that the modern State has its “enormous strength and depth.” It is
the typical mind, the very essence of reason, whose completeness is
directly as the completeness of each of its terms or sides or factors;
and secure in the logical confidence that feeling and self-
consciousness, the more they attain their fulness, must return the
more certainly to their place in the reasonable system which is their
very nature. As ultimate power, the State maintains on one side the
attitude of an external necessity towards the spheres of private life,
of the family, and of the economic world. It may intervene by force
to remove hindrances in the path of the common good, which
accident and immaturity may have placed there. But, in its essence,
the State is the indwelling and explicit end of these modes of living,
and is strong in its union of the universal purpose with the particular
interests of mankind. It is, in short, the incarnation of the general or](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-75-320.jpg)
![as if there were such a thing as “the people” over and above the
organised means of expressing and adjusting the will of the
community—we are saying what is, strictly speaking, meaningless. It
is just the point of difference between Rousseau’s two views. We
saw that Rousseau clearly explained the impossibility of expressing
the general {283} will except by a determinate system of law. But
what he seemed to suggest, and was taken to mean, by popular
Sovereignty, was no doubt just the view which Hegel condemns. It is
essentially the same question as how a constitution can be made.
Strictly, a constitution cannot be made except by modification of an
existing constitution. If, to put a case, you have a multitude new to
each other in some extra-political colony, they must assume a
constitution, so to speak, before they can make one. Law and
constitution are utterances of the spirit of a nation.
The form of State which Hegel analyses is a modern constitutional
monarchy, with an executive (ministers sitting in the chambers, as
he is careful to urge) and Chambers or Estates representing the
classes developed in the civic community. Representation, he urges,
is of bodies or interests rather than of masses of individuals, and the
Corporations or Trade Societies have also an important place directly,
by their touch with the departments of the executive government.
[1] The general principle is, as indicated above, that the problems of
connection between considerable particular interests and the
universal interests of the community are, so to speak, prepared on
the ground of the Corporation and Bourgeois Society for a solution in
the interest of the common good by the Legislative and the
Executive Government.
[1] Much as through inspectors and commissions the opinion of
Trade Unions, Friendly Societies, and Co-operators is elicited by our](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-77-320.jpg)
![{285} individuals as parts in a connected structure, which unite in
willing a certain type of life as a common good in which they find
their own. It has the same content as that of Religion; but in an
explicit and rationalised form as contrasted with the form of feeling.
Only the separation of Church and State, and the division of the
Churches against one another, have made it possible for the State to
exhibit its own free and ethical character in true fulness, apart from
both dogmatic authority and anarchic fanaticism.
9. Publicity of discussion in the assembly of the classes or estates
is the great means of civic education. It is not in the least true that
every one knows what is for the good of the State, and has only to
go down to the House and utter it. It is in the work of expression [1]
and discussion that the good takes form by adjustment of private
views to facts and needs brought to bear by criticism. “The views a
man plumes himself on when he is at home with his wife and friends
are one thing; it is quite another thing what happens in a great
assembly, where one shrewd idea devours the other.” [2]
[1] It is a remarkable point in English politics to-day that
legislation is practically in the hands of the Government
departments. Bills are rejected or “knocked about in Committee”;
but the mass of organised knowledge necessary to initiate legislation
in a complex society can hardly be found outside the gathered
experience of an office which has continuity in dealing with the same
problems. This tendency more than justifies Hegel’s point of view.
An act of the “General Will” has not only, as he said, to be moulded
by running the gauntlet of public and critical discussion, but has
even to be first drafted by the help of immense piles of experience,
which the general mind does not possess, and could not deal with,](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-79-320.jpg)
![but which, nevertheless, enable its typical wish and intention to be
embodied in effective form.
[2] Rechtsphil., sect. 315.
{286} The free judgment of individuals based on the publicity of
political discussion is “public opinion.” In public opinion we have an
actual existent contradiction. As public, it is sound and true, and
contains the ethical spirit of the State. As expressed by individuals in
their particular judgments, on which they plume themselves, it is full
of falsehood and vanity. It is the bad which is peculiar, and which
people pride themselves on; the rational is universal in its nature,
though not necessarily common. Public opinion is a contradictory
appearance, in which the true exists as false. It is no accident, but
inevitable insight, that leads both of these characters to be
proverbially expressed, as in “Vox populi, vox Dei,” contrasted with
Ariosto’s
“Che’l Volgare ignorante ogn’un’ riprenda
E parli plu di qual che meno intenda”; [1]
or Goethe’s
“Zuschlagen kann die Masse
Da ist sie respektabel;
Urtheilen gelingt ihr miserabel.” [2]
or the “mostly fools” of Carlyle.
[1] “That the ignorant vulgar reproves everyone, and talks most of
what it understands least.”](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-80-320.jpg)
![[2] “The masses are respectable hands at fighting, but miserable
hands at judging.”
Now, as public opinion thus combines truth and falsehood, the
public cannot be in earnest with both, i.e. both cannot be its real
will. But if we restrict ourselves to its express utterance, we cannot
possibly tell what it is in earnest with—because it does not know.
Therefore, the degree of passion {287} with which a given opinion is
maintained throws no light on the question, on what points the
public is really in earnest, in the sense of the “real will.” This can
only be known from the substantive reality, which is the “true
inwardness” of public opinion. This substantive reality, the true
merits of any case, is not to be got by the study of mere public
opinion as expressed, but when it is successfully divined and
asserted, public opinion will always come round to it. If we ask how
it is to be divined or known, we must go back to the analogy of a
theory. The solution must be constructed so as to satisfy the real
facts or needs, and the real facts or needs only become known in
proportion as it is constructed, just as in scientific discovery. The
man who can see and do what his age wills and demands is the
great man of the age. Public opinion, then, demands to be at once
esteemed and contemned; esteemed in its essential basis,
contemned in its conscious expression. It is, however, the principle
of the modern world that every one is allowed to contribute his
opinion. When he has contributed it, and so far satisfied the impulse
of self-assertion, he is likely to acquiesce in what is done, to which,
he can feel, he has thrown in some element of suggestion or
criticism.
10. In concluding this chapter, we will attempt to estimate the
nearness of such an analysis of the State to the actual facts of life,](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-81-320.jpg)
![admitting certain appearances against it, but rejecting pessimistic
views which rest on false abstractions.
I will state the difficulties as they appeared to T.H. Green, a
cautious and practical Englishman, {288} well experienced in local
politics, and acquainted with different classes of men. [1]
“To an Athenian slave, who might be used to gratify a master’s
lust, it would have been a mockery to speak of the State as a
realisation of freedom; and perhaps it would not be much less to
speak of it as such to an untaught and under-fed denizen of a
London yard with gin shops on the right hand and on the left.”
“It is true that the necessity which the State lays on the
individual is for the most part one to which he is so accustomed
that he no longer kicks against it; but what is it, we may ask, but
an external necessity, which he no more lays on himself than he
does the weight of the atmosphere or the pressure of summer
heat and winter frosts, that compels the ordinary citizen to pay
rates and taxes, to serve in the army, to abstain from walking
over the Squire’s fields, snaring his hares, or fishing his preserved
streams, to pay his rent, to respect those artificial rights of
property which only the possessors of them have any obvious
interest in maintaining, or even (if he is one of the proletariate) to
keep his hands off the superfluous wealth of his neighbour when
he has none of his own to lose?”
“A conception does not float in the air. It must be somebody’s
conception. Whose conception, then, of general good is it that
these institutions represent?”](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-82-320.jpg)
![“Is it not seriously misleading, when the requirements of the
State have so largely arisen out of force directed by selfish
motives, and when the motive of obedience to these
requirements is determined by fear, to {289} speak of them as
having a common source with the morality of which it is admitted
that the essence is to be disinterested and spontaneous?”
[1] Principles of Political Obligation, p. 8; cf. p. 127 ff.
I have quoted these passages—the whole section should be
carefully read—in order to state plainly a paradox which affects the
theory of society from beginning to end. It continually shows itself in
the pessimistic criticism of economic motive, political motive, and of
every-day social motive.
The whole question really depends on our understanding of the
relation of abstract and concrete. It is plain, as Green says, that the
idea of a common good has never been the sole influence operative
in the formation or maintenance of States. And, in as far as it has
operated at all, it has only done so in very imperfect forms. Green
goes so far as to say that Hegel’s account of freedom as realised in
the State does not seem to correspond to the facts of society as it is,
or even as, under the unalterable conditions of human nature, it
ever could be; though, no doubt, there is a work of moral liberation,
which society, through its various agencies, is constantly carrying on
for the individual.
Now, the truth of these criticisms may be granted in the same
sense in which we grant the imperfection of knowledge (as currently
conceived) or of morality—imperfections not accidental, but inherent
in each particular form of human experience. The conflict of
interests, the failure to reconcile rights, and the weight and](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-83-320.jpg)
![opaqueness, so to speak, of law and custom to the individual mind,
are contradictions of the same type and {290} due to causes of the
same kind as those which arise in the world of ethics and of theory.
And, though the new relations which spring up in society are
perpetually resulting in new contradictions, there is no reason to
compare the State unfavourably, in this respect, with Morality or with
Science. The contradictions, in fact, are the material of organisation.
[1]
[1] Take, for instance, the chaos of the medical charities of
London. It consists of endeavours to adjust help to needs, which
endeavours are themselves unadjusted to each other. Thus, precisely
as in the theoretical progress, the unadjustment of adjustments
brings out ever new contradictions which demand readjustment.
Without differing profoundly from Green in theory, therefore, we
venture to assign a greatly diminished importance to his criticisms.
This is due in part to the growth of a more intimate experience,
owing in some measure to his initiative, which seems to show the
essentials of life to be far more identical throughout the so-called
classes of society than is admitted by such a passage as that cited
above about the dweller in a London yard. [1] It is due, further, and
in connection with such experience, to the psychological conceptions
developed in previous chapters, according to which the place of
actual fear of punishment in maintaining the social system is really
very small, while {291} the place of a habituation, which is
essentially ethical, is comparatively large. These suggestions, which
lead us to lay decreasing stress on Green’s criticism of Hegel, point
wholly in the general direction of his own convictions, and we may
finally meet the general difficulty, which expresses itself in](https://image.slidesharecdn.com/65956-250510122630-c1ce45f7/85/High-Performance-Networks-From-Supercomputing-to-Cloud-Computing-1st-Edition-Dennis-Abts-84-320.jpg)
High Performance Networks From Supercomputing to Cloud Computing 1st Edition Dennis Abts
- 1. High Performance Networks From Supercomputing to Cloud Computing 1st Edition Dennis Abts download https://ebookgate.com/product/high-performance-networks-from- supercomputing-to-cloud-computing-1st-edition-dennis-abts/ Get Instant Ebook Downloads – Browse at https://ebookgate.com
- 2. Get Your Digital Files Instantly: PDF, ePub, MOBI and More Quick Digital Downloads: PDF, ePub, MOBI and Other Formats High Performance and Hardware Aware Computing Rainer Buchty https://ebookgate.com/product/high-performance-and-hardware- aware-computing-rainer-buchty/ High performance communication networks 2nd Edition Jean Walrand https://ebookgate.com/product/high-performance-communication- networks-2nd-edition-jean-walrand/ Working in Teams Moving From High Potential to High Performance 1st Edition Brian A. Griffith https://ebookgate.com/product/working-in-teams-moving-from-high- potential-to-high-performance-1st-edition-brian-a-griffith/ Cloud Computing Bible 1st Edition Barrie Sosinsky https://ebookgate.com/product/cloud-computing-bible-1st-edition- barrie-sosinsky/
- 3. Phase Locking in High Performance Systems From Devices to Architectures 1st Edition Behzad Razavi https://ebookgate.com/product/phase-locking-in-high-performance- systems-from-devices-to-architectures-1st-edition-behzad-razavi/ High Performance Computing Programming and Applications Chapman Hall CRC Computational Science 1st Edition John Levesque https://ebookgate.com/product/high-performance-computing- programming-and-applications-chapman-hall-crc-computational- science-1st-edition-john-levesque/ Biomedical Diagnostics and Clinical Technologies Applying High Performance Cluster and Grid Computing 1st Edition Manuela Pereira https://ebookgate.com/product/biomedical-diagnostics-and- clinical-technologies-applying-high-performance-cluster-and-grid- computing-1st-edition-manuela-pereira/ Architecting the Cloud Design Decisions for Cloud Computing Service Models 1st Edition Michael J. Kavis https://ebookgate.com/product/architecting-the-cloud-design- decisions-for-cloud-computing-service-models-1st-edition-michael- j-kavis/ Practical Guide to High Performance Engineering Plastics 1st Edition Kemmish https://ebookgate.com/product/practical-guide-to-high- performance-engineering-plastics-1st-edition-kemmish/
- 4. Morgan Claypool Publishers & w w w . m o r g a n c l a y p o o l . c o m Series Editor: Mark D. Hill, University of Wisconsin MOR GAN & CL AYPOOL C M & Morgan Claypool Publishers & About SYNTHESIs This volume is a printed version of a work that appears in the Synthesis Digital Library of Engineering and Computer Science. Synthesis Lectures provide concise,original presentations of important research and development topics, published quickly, in digital and print formats. For more information visit www.morganclaypool.com SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE Mark D. Hill, Series Editor ISBN: 978-1-60845-402-0 9 781608 454020 90000 Series ISSN: 1935-3235 SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE HIGH PERFORMANCE DATACENTER NETWORKS ABTS • KIM High Performance Datacenter Networks Architectures, Algorithms, and Opportunity Dennis Abts, Google Inc. and John Kim, Korea Advanced Institute of Sceince and Technology Datacenter networks provide the communication substrate for large parallel computer systems that form the ecosystem for high performance computing (HPC) systems and modern Internet appli- cations. The design of new datacenter networks is motivated by an array of applications ranging from communication intensive climatology, complex material simulations and molecular dynamics to such Internet applications as Web search, language translation, collaborative Internet applications, streaming video and voice-over-IP. For both Supercomputing and Cloud Computing the network enables distributed applications to communicate and interoperate in an orchestrated and efficient way. This book describes the design and engineering tradeoffs of datacenter networks. It describes interconnection networks from topology and network architecture to routing algorithms,and presents opportunities for taking advantage of the emerging technology trends that are influencing router microarchitecture. With the emergence of “many-core”processor chips, it is evident that we will also need “many-port” routing chips to provide a bandwidth-rich network to avoid the performance limiting effects of Amdahl’s Law. We provide an overview of conventional topologies and their routing algorithms and show how technology, signaling rates and cost-effective optics are motivating new network topologies that scale up to millions of hosts. The book also provides detailed case studies of two high performance parallel computer systems and their networks. High Performance Datacenter Networks Architectures, Algorithms, and Opportunity Dennis Abts John Kim Morgan Claypool Publishers & w w w . m o r g a n c l a y p o o l . c o m Series Editor: Mark D. Hill, University of Wisconsin MOR GAN & CL AYPOOL C M & Morgan Claypool Publishers & About SYNTHESIs This volume is a printed version of a work that appears in the Synthesis Digital Library of Engineering and Computer Science. Synthesis Lectures provide concise,original presentations of important research and development topics, published quickly, in digital and print formats. For more information visit www.morganclaypool.com SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE Mark D. Hill, Series Editor ISBN: 978-1-60845-402-0 9 781608 454020 90000 Series ISSN: 1935-3235 SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE HIGH PERFORMANCE DATACENTER NETWORKS ABTS • KIM High Performance Datacenter Networks Architectures, Algorithms, and Opportunity Dennis Abts, Google Inc. and John Kim, Korea Advanced Institute of Sceince and Technology Datacenter networks provide the communication substrate for large parallel computer systems that form the ecosystem for high performance computing (HPC) systems and modern Internet appli- cations. The design of new datacenter networks is motivated by an array of applications ranging from communication intensive climatology, complex material simulations and molecular dynamics to such Internet applications as Web search, language translation, collaborative Internet applications, streaming video and voice-over-IP. For both Supercomputing and Cloud Computing the network enables distributed applications to communicate and interoperate in an orchestrated and efficient way. This book describes the design and engineering tradeoffs of datacenter networks. It describes interconnection networks from topology and network architecture to routing algorithms,and presents opportunities for taking advantage of the emerging technology trends that are influencing router microarchitecture. With the emergence of “many-core”processor chips, it is evident that we will also need “many-port” routing chips to provide a bandwidth-rich network to avoid the performance limiting effects of Amdahl’s Law. We provide an overview of conventional topologies and their routing algorithms and show how technology, signaling rates and cost-effective optics are motivating new network topologies that scale up to millions of hosts. The book also provides detailed case studies of two high performance parallel computer systems and their networks. High Performance Datacenter Networks Architectures, Algorithms, and Opportunity Dennis Abts John Kim Morgan Claypool Publishers & w w w . m o r g a n c l a y p o o l . c o m Series Editor: Mark D. Hill, University of Wisconsin MOR GAN & CL AYPOOL C M & Morgan Claypool Publishers & About SYNTHESIs This volume is a printed version of a work that appears in the Synthesis Digital Library of Engineering and Computer Science. Synthesis Lectures provide concise,original presentations of important research and development topics, published quickly, in digital and print formats. For more information visit www.morganclaypool.com SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE Mark D. Hill, Series Editor ISBN: 978-1-60845-402-0 9 781608 454020 90000 Series ISSN: 1935-3235 SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE HIGH PERFORMANCE DATACENTER NETWORKS ABTS • KIM High Performance Datacenter Networks Architectures, Algorithms, and Opportunity Dennis Abts, Google Inc. and John Kim, Korea Advanced Institute of Sceince and Technology Datacenter networks provide the communication substrate for large parallel computer systems that form the ecosystem for high performance computing (HPC) systems and modern Internet appli- cations. The design of new datacenter networks is motivated by an array of applications ranging from communication intensive climatology, complex material simulations and molecular dynamics to such Internet applications as Web search, language translation, collaborative Internet applications, streaming video and voice-over-IP. For both Supercomputing and Cloud Computing the network enables distributed applications to communicate and interoperate in an orchestrated and efficient way. This book describes the design and engineering tradeoffs of datacenter networks. It describes interconnection networks from topology and network architecture to routing algorithms,and presents opportunities for taking advantage of the emerging technology trends that are influencing router microarchitecture. With the emergence of “many-core”processor chips, it is evident that we will also need “many-port” routing chips to provide a bandwidth-rich network to avoid the performance limiting effects of Amdahl’s Law. We provide an overview of conventional topologies and their routing algorithms and show how technology, signaling rates and cost-effective optics are motivating new network topologies that scale up to millions of hosts. The book also provides detailed case studies of two high performance parallel computer systems and their networks. High Performance Datacenter Networks Architectures, Algorithms, and Opportunity Dennis Abts John Kim
- 5. High Performance Datacenter Networks Architectures, Algorithms, and Opportunities
- 6. Synthesis Lectures on Computer Architecture Editor Mark D. Hill, University of Wisconsin Synthesis Lectures on Computer Architecture publishes 50- to 100-page publications on topics pertaining to the science and art of designing, analyzing, selecting and interconnecting hardware components to create computers that meet functional, performance and cost goals. The scope will largely follow the purview of premier computer architecture conferences, such as ISCA, HPCA, MICRO, and ASPLOS. High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities Dennis Abts and John Kim 2011 Quantum Computing for Architects, Second Edition Tzvetan Metodi, Fred Chong, and Arvin Faruque 2011 Processor Microarchitecture: An Implementation Perspective Antonio González, Fernando Latorre, and Grigorios Magklis 2010 Transactional Memory, 2nd edition Tim Harris, James Larus, and Ravi Rajwar 2010 Computer Architecture Performance Evaluation Methods Lieven Eeckhout 2010 Introduction to Reconfigurable Supercomputing Marco Lanzagorta, Stephen Bique, and Robert Rosenberg 2009 On-Chip Networks Natalie Enright Jerger and Li-Shiuan Peh 2009
- 7. iii The Memory System: You Can’t Avoid It, You Can’t Ignore It, You Can’t Fake It Bruce Jacob 2009 Fault Tolerant Computer Architecture Daniel J. Sorin 2009 The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines free access Luiz André Barroso and Urs Hölzle 2009 Computer Architecture Techniques for Power-Efficiency Stefanos Kaxiras and Margaret Martonosi 2008 Chip Multiprocessor Architecture: Techniques to Improve Throughput and Latency Kunle Olukotun, Lance Hammond, and James Laudon 2007 Transactional Memory James R. Larus and Ravi Rajwar 2006 Quantum Computing for Computer Architects Tzvetan S. Metodi and Frederic T. Chong 2006
- 8. Copyright © 2011 by Morgan & Claypool All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—electronic, mechanical, photocopy, recording, or any other except for brief quotations in printed reviews, without the prior permission of the publisher. High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities Dennis Abts and John Kim www.morganclaypool.com ISBN: 9781608454020 paperback ISBN: 9781608454037 ebook DOI 10.2200/S00341ED1V01Y201103CAC014 A Publication in the Morgan & Claypool Publishers series SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE Lecture #14 Series Editor: Mark D. Hill, University of Wisconsin Series ISSN Synthesis Lectures on Computer Architecture Print 1935-3235 Electronic 1935-3243
- 9. High Performance Datacenter Networks Architectures, Algorithms, and Opportunities Dennis Abts Google Inc. John Kim Korea Advanced Institute of Science and Technology (KAIST) SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE #14 C M & cLaypool Morgan publishers &
- 10. ABSTRACT Datacenter networks provide the communication substrate for large parallel computer systems that form the ecosystem for high performance computing (HPC) systems and modern Internet appli- cations. The design of new datacenter networks is motivated by an array of applications ranging from communication intensive climatology, complex material simulations and molecular dynamics to such Internet applications as Web search,language translation,collaborative Internet applications, streaming video and voice-over-IP. For both Supercomputing and Cloud Computing the network enables distributed applications to communicate and interoperate in an orchestrated and efficient way. This book describes the design and engineering tradeoffs of datacenter networks. It de- scribes interconnection networks from topology and network architecture to routing algorithms, and presents opportunities for taking advantage of the emerging technology trends that are influ- encing router microarchitecture. With the emergence of “many-core” processor chips, it is evident that we will also need “many-port” routing chips to provide a bandwidth-rich network to avoid the performance limiting effects of Amdahl’s Law. We provide an overview of conventional topologies and their routing algorithms and show how technology, signaling rates and cost-effective optics are motivating new network topologies that scale up to millions of hosts.The book also provides detailed case studies of two high performance parallel computer systems and their networks. KEYWORDS network architecture and design, topology, interconnection networks, fiber optics, par- allel computer architecture, system design
- 11. vii Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Note to the Reader. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 1.1 From Supercomputing to Cloud Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Beowulf: The Cluster is Born . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Overview of Parallel Programming Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4 Putting it all together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.5 Quality of Service (QoS) requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.6 Flow control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.6.1 Lossy flow control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.6.2 Lossless flow control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.7 The rise of ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.1 Interconnection networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2 Technology trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.3 Topology, Routing and Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.4 Communication Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Topology Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.2 Types of Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.3 Mesh, Torus, and Hypercubes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.3.1 Node identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.3.2 k-ary n-cube tradeoffs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
- 12. viii 4 High-Radix Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.1 Towards High-radix Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.2 Technology Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.1 Pin Bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.2 Economical Optical Signaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.3 High-Radix Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.3.1 High-Dimension Hypercube, Mesh, Torus . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.3.2 Butterfly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.3.3 High-Radix Folded-Clos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.3.4 Flattened Butterfly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.3.5 Dragonfly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.3.6 HyperX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5 Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1 Routing Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.1 Objectives of a Routing Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 5.2 Minimal Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 5.2.1 Deterministic Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 5.2.2 Oblivious Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.3 Non-minimal Routing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.3.1 Valiant’s algorithm (VAL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.3.2 Universal Global Adaptive Load-Balancing (UGAL) . . . . . . . . . . . . . . . . 42 5.3.3 Progressive Adaptive Routing (PAR) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.3.4 Dimensionally-Adaptive, Load-balanced (DAL) Routing . . . . . . . . . . . . . 43 5.4 Indirect Adaptive Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.5 Routing Algorithm Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 5.5.1 Example 1: Folded-Clos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 5.5.2 Example 2: Flattened Butterfly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 5.5.3 Example 3: Dragonfly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 6 Scalable Switch Microarchitecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 6.1 Router Microarchitecture Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 6.2 Scaling baseline microarchitecture to high radix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 6.3 Fully Buffered Crossbar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 6.4 Hierarchical Crossbar Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 6.5 Examples of High-Radix Routers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
- 13. ix 6.5.1 Cray YARC Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 6.5.2 Mellanox InfiniScale IV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 7 System Packaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 7.1 Packaging hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 7.2 Power delivery and cooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 7.3 Topology and Packaging Locality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 8 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 8.1 Cray BlackWidow Multiprocessor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 8.1.1 BlackWidow Node Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 8.1.2 High-radix Folded-Clos Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 8.1.3 System Packaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 8.1.4 High-radix Fat-tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 8.1.5 Packet Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 8.1.6 Network Layer Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 8.1.7 Data-link Layer Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 8.1.8 Serializer/Deserializer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 8.2 Cray XT Multiprocessor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 8.2.1 3-D torus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 8.2.2 Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 8.2.3 Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 8.2.4 SeaStar Router Microarchitecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 8.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 9 Closing Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 9.1 Programming models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 9.2 Wire protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 9.3 Opportunities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Authors’ Biographies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
- 15. Preface This book is aimed at the researcher, graduate student and practitioner alike. We provide some background and motivation to provide the reader with a substrate upon which we can build the new concepts that are driving high-performance networking in both supercomputing and cloud computing. We assume the reader is familiar with computer architecture and basic networking concepts. We show the evolution of high-performance interconnection networks over the span of two decades, and the underlying technology trends driving these changes. We describe how to apply these technology drivers to enable new network topologies and routing algorithms that scale to millions of processing cores. We hope that practitioners will find the material useful for making design tradeoffs, and researchers will find the material both timely and relevant to modern parallel computer systems which make up today’s datacenters. Dennis Abts and John Kim March 2011
- 17. Acknowledgments While we draw from our experience at Cray and Google and academic work on the design and operation of interconnection networks, most of what we learned is the result of hard work, and years of experience that have led to practical insights. Our experience benefited tremendously from our colleagues Steve Scott at Cray, and Bill Dally at Stanford University. In addition, many hours of whiteboard-huddled conversations with Mike Marty, Philip Wells, Hong Liu, and Peter Klausler at Google. We would also like to thank Google colleagues James Laudon, Bob Felderman, Luiz Barroso, and Urs Hölzle for reviewing draft versions of the manuscript. We want to thank the reviewers, especially Amin Vahdat and Mark Hill for taking the time to carefully read and provide feedback on early versions of this manuscript. Thanks to Urs Hölzle for guidance, and Kristin Weissman at Google and Michael Morgan at Morgan & Claypool Publishers. Finally, we are grateful for Mark Hill and Michael Morgan for inviting us to this project and being patient with deadlines. Finally, and most importantly, we would like to thank our loving family members who gra- ciously supported this work and patiently allowed us to spend our free time to work on this project. Without their enduring patience and with an equal amount of prodding, this work would not have materialized. Dennis Abts and John Kim March 2011
- 19. Note to the Reader We very much appreciate any feedback, suggestions, and corrections you might have on our manuscript. The Morgan & Claypool publishing process allows a lightweight method to revise the electronic edition. We plan to revise the manuscript relatively often, and will gratefully acknowledge any input that will help us to improve the accuracy, readability, or general usefulness of the book. Please leave your feedback at http://tinyurl.com/HPNFeedback Dennis Abts and John Kim March 2011
- 21. 1 C H A P T E R 1 Introduction Today’s datacenters have emerged from the collection of loosely connected workstations, which shaped the humble beginnings of the Internet, and grown into massive “warehouse-scale comput- ers” (Figure 1.1) capable of running the most demanding workloads.Barroso and Hölzle describe the architecture of a warehouse-scale computer (WSC) [9] and give an overview of the programming model and common workloads executed on these machines.The hardware building blocks are pack- aged into “racks” of about 40 servers, and many racks are interconnected using a high-performance network to form a “cluster” with hundreds or thousands of tightly-coupled servers for performance, cooling towers power substation warehouse-scale computer Figure 1.1: A datacenter with cooling infrastructure and power delivery highlighted.
- 22. 2 1. INTRODUCTION ! ! Figure 1.2: Comparison of web search interest and terminology. but loosely-coupled for fault tolerance and isolation. This highlights some distinctions between what have traditionally been called“supercomputers” and what we now consider“cloud computing,” which appears to have emerged around 2008 (based on the relative Web Search interest shown in Figure 1.2) as a moniker for server-side computing. Increasingly, our computing needs are moving away from desktop computers toward more mobile clients (e.g., smart phones, tablet computers, and net- books) that depend on Internet services, applications, and storage. As an example, it is much more efficient to maintain a repository of digital photography on a server in the “cloud” than on a PC-like computer that is perhaps not as well maintained as a server in a large datacenter, which is more reminiscent of a clean room environment than a living room where your precious digital memories are subjected to the daily routine of kids, spills, power failures, and varying temperatures; in addition, most consumers upgrade computers every few years,requiring them to migrate all their precious data to their newest piece of technology. In contrast, the “cloud” provides a clean, temperature controlled environment with ample power distribution and backup. Not to mention your data in the “cloud” is probably replicated for redundancy in the event of a hardware failure the user data is replicated and restored generally without the user even aware that an error occurred.
- 23. 1.1. FROM SUPERCOMPUTING TO CLOUD COMPUTING 3 1.1 FROM SUPERCOMPUTING TO CLOUD COMPUTING As the ARPANET transformed into the Internet over the past forty years, and the World Wide Web emerges from adolescence and turns twenty, this metamorphosis has seen changes in both supercomputing and cloud computing. The supercomputing industry was born in 1976 when Sey- mour Cray announced the Cray-1 [54]. Among the many innovations were its processor design, process technology, system packaging, and instruction set architecture. The foundation of the ar- chitecture was based on the notion of vector operations that allowed a single instruction to operate on an array, or “vector,” of elements simultaneously. In contrast to scalar processors of the time whose instructions operated on single data items. The vector parallelism approach dominated the high-performance computing landscape for much of the 1980s and early 1990s until “commodity” microprocessors began aggressively implementing forms of instruction-level parallelism (ILP) and better cache memory systems to exploit spatial and temporal locality exhibited by most applications. Improvements in CMOS process technology and full-custom CMOS design practices allowed mi- croprocessors to quickly ramp up clock rates to several gigahertz. This coupled with multi-issue pipelines; efficient branch prediction and speculation eventually allowed microprocessors to catch up with their proprietary vector processors from Cray, Convex, and NEC. Over time, conventional microprocessors incorporated short vector units (e.g., SSE, MMX, AltiVec) into the instruction set. However, the largest beneficiary of vector processing has been multimedia applications as evidenced by the jointly developed (by Sony,Toshiba,and IBM) Cell processor which found widespread success in Sony’s Playstation3 game console, and even some special-purpose computer systems like Mercury Systems. Parallel applications eventually have to synchronize and communicate among parallel threads. Amdahl’s Law is relentless and unless enough parallelism is exposed,the time spent orchestrating the parallelism and executing the sequential region will ultimately limit the application performance [27]. 1.2 BEOWULF: THE CLUSTER IS BORN In 1994 Thomas Sterling (then dually affiliated with the California Institute of Technology and NASAs JPL) and Donald Becker (then a researcher at NASA) assembled a parallel computer that became known as a Beowulf cluster1. What was unique about Beowulf [61] systems was that they were built from common “off-the-shelf” computers, as Figure 1.3 shows, system packaging was not an emphasis. More importantly, as a loosely-coupled distributed memory machine, Beowulf forced researchers to think about how to efficiently program parallel computers. As a result, we benefited from portable and free programming interfaces such as parallel virtual machines (PVM), message passing interfaces (MPICH and OpenMPI), local area multiprocessor (LAM); with MPI being embraced by the HPC community and highly optimized. The Beowulf cluster was organized so that one machine was designated the “server,” and it managed job scheduling, pushing binaries to clients, and monitoring. It also acted as the gateway 1The genesis of the name comes from the poem which describes Beowulf as having “thirty men’s heft of grasp in the gripe of his hand.”
- 24. 4 1. INTRODUCTION Figure 1.3: An 128 processor Beowulf cluster at NASA. to the “outside world,” so researchers had a login host. The model is still quite common: with some nodes being designated as service and IO nodes where users actually login to the parallel machine. From there, they can compile their code, and launch the job on “compute only” nodes — the worker bees of the colony — and console information, machine status is communicated to the service nodes. 1.3 OVERVIEW OF PARALLEL PROGRAMMING MODELS Early supercomputers were able to work efficiently, in part, because they shared a common physical memory space. As a result, communication among processors was very efficient as they updated shared variables and operated on common data. However, as the size of the systems grew, this shared memory model evolved into a distributed shared memory (DSM) model where each processing node owns a portion of the machines physical memory and the programmer is provided with a logically shared address space making it easy to reason about how the application is partitioned and communication among threads. The Stanford DASH [45] was the first to demonstrate this cache- coherent non-uniform memory (ccNUMA) access model, and the SGI Origin2000 [43] was the first machine to successfully commercialize the DSM architecture. We commonly refer to distributed memory machines as“clusters” since they are loosely-coupled and rely on message passing for communication among processing nodes. With the inception of Beowulf clusters, the HPC community realized they could build modest-sized parallel computers on
- 25. 1.4. PUTTING IT ALL TOGETHER 5 a relatively small budget. To their benefit, the common benchmark for measuring the performance of a parallel computer is LINPACK, which is not communication intensive, so it was commonplace to use inexpensive Ethernet networks to string together commodity nodes. As a result, Ethernet got a foothold on the list of the TOP500 [62] civilian supercomputers with almost 50% of the TOP500 systems using Ethernet. 1.4 PUTTING IT ALL TOGETHER The first Cray-1 [54] supercomputer had expected to ship one system per quarter in 1977. Today, microprocessor companies have refined their CMOS processes and manufacturing making them very cost-effective building blocks for large-scale parallel systems capable of 10s of petaflops. This shift away from “proprietary” processors and trend toward “commodity” processors has fueled the growth of systems. At the time of this writing, the largest computer on the TOP500 list [62] has in excess of 220,000 cores (see Figure 7.5) and consumes almost seven megawatts! A datacenter server has many commonalities as one used in a supercomputer, however, there are also some very glaring differences. We enumerate several properties of both a warehouse-scale computer (WSC) and a supercomputer (Cray XE6). Datacenter server • Sockets per server 2 sockets x86 platform • Memory capacity 16 GB DRAM • Disk capacity 5×1TB disk drive, and 1×160GB SSD (FLASH) • Compute density 80 sockets per rack • Network bandwidth per rack 1×48-port GigE switch with 40 down links, and 8 uplinks (5× oversubscription) • Network bandwidth per socket 100 Mb/s if 1 GigE rack switch, or 1 Gb/s if 10 GigE rack switch Supercomputer server • Sockets per server 8 sockets x86 platform • Memory capacity 32 or 64 GB DRAM • Disk capacity IO capacity varies. Each XIO blade has four PCIe-Gen2 interfaces, for a total of 96 PCIe-Gen2 ×16 IO devices for a peak IO bandwidth of 768 GB/s per direction. • Compute density 192 sockets per rack
- 26. 6 1. INTRODUCTION • Networkbandwidthperrack 48×48-port Gemini switch chips each with 160 GB/s switching bandwidth • Network bandwidth per socket 9.6GB/s injection bandwidth with non-coherent Hyper- Transport 3.0 (ncHT3) Several things stand out as differences between a datacenter server and supercomputer node. First, the compute density for the supercomputer is significantly better than a standard 40U rack. On the other hand, this dense packaging also puts pressure on cooling requirements not to mention power delivery. As power and its associated delivery become increasingly expensive, it becomes more important to optimize the number of operations per watt; often the size of a system is limited by power distribution and cooling infrastructure. AnotherpointisthevastdifferenceinnetworkbandwidthpersocketinlargepartbecausencHT3 is a much higher bandwidth processor interface than PCIe-Gen2, however, as PCI-Gen3×16 be- comes available we expect that gap to narrow. 1.5 QUALITY OF SERVICE (QOS) REQUIREMENTS With HPC systems it is commonplace to dedicate the system for the duration of application ex- ecution. Allowing all processors to be used for compute resources. As a result, there is no need for performance isolation from competing applications. Quality of Service (QoS) provides both per- formance isolation and differentiated service for applications2. Cloud computing often has a varied workloads requiring multiple applications to share resources. Workload consolidation [33] is becom- ing increasingly important as memory and processor cost increase, as a result so does the value of increased system utilization. The QoS class refers to the end-to-end class of service as observed by the application. In principle, QoS is divided into three categories: Best effort - traffic is treated as a FIFO with no differentiation provided. Differentiated service - also referred to as “soft QoS” where traffic is given a statistical preference over other traffic. This means it is less likely to be dropped relative to best effort traffic, for example, resulting in lower average latency and increased average bandwidth. Guaranteed service - also referred to as “hard QoS” where a fraction of the network bandwidth is reserved to provide no-loss, low jitter bandwidth guarantees. In practice,there are many intermediate pieces which are,in part,responsible for implementing a QoS scheme. A routing algorithm determines the set of usable paths through the network between any source and destination. Generally speaking, routing is a background process that attempts to load- balance the physical links in the system taking into account any network faults and programming 2We use the term “applications” loosely here to represent processes or threads, at whatever granularity a service level agreement is applied.
- 27. 1.6. FLOW CONTROL 7 the forwarding tables within each router. When a new packet arrives, the header is inspected and the network address of the destination is used to index into the forwarding table which emits the output port where the packet is scheduled for transmission.The “packet forwarding” process is done on a packet-by-packet basis and is responsible for identifying packets marked for special treatment according to its QoS class. The basic unit over which a QoS class is applied is the flow. A flow is described as a tuple (SourceIP, SourcePort, DestIP, DestPort). Packets are marked by the host or edge switch using either 1) port range, or 2) host (sender/client-side) marking. Since we are talking about end-to-end service levels, ideally the host which initiates the communication would request a specific level of service. This requires some client-side API for establishing the QoS requirements prior to sending a message. Alternatively, edge routers can mark packets as they are injected into the core fabric. Packets are marked with their service class which is interpreted at each hop and acted upon by routers along the path.For common Internet protocols,the differentiated service (DS) field of the IP header provides this function as defined by the DiffServ [RFC2475] architecture for network layer QoS. For compatibility reasons, this is the same field as the type of service (ToS) field [RFC791] of the IP header. Since the RFC does not clearly describe how “low,” “medium,” or “high” are supposed to be interpreted, it is common to use five classes: best effort (BE), AF1, AF2, AF3, AF4, and set the drop priority to 0 (ignored). 1.6 FLOW CONTROL Surprisingly, a key difference in system interconnects is flow control. How the switch and buffer resources are managed is very different in Ethernet than what is typical in a supercomputer in- terconnect. There are several kinds of flow control in a large distributed parallel computer. The interconnection network is a shared resource among all the compute nodes, and network resources must be carefully managed to avoid corrupting data, overflowing a buffer, etc.The basic mechanism by which resources in the network are managed is flow control. Flow control provides a simple ac- counting method for managing resources that are in demand by multiple uncoordinated sources. The resource is managed in units of flits (flow control units). When a resource is requested but not currently available for use, we must decide what to do with the incoming request. In general, we can 1) drop the request and all subsequent requests until the resource is freed, or 2) block and wait for the request to free. 1.6.1 LOSSY FLOW CONTROL With a lossy flow control [20, 48], the hardware can discard packets until there is room in the desired resource. This approach is usually applied to input buffers on each switch chip, but also applies to resources in the network interface controller (NIC) chip as well. When packets are dropped, the software layers must detect the loss, usually through an unexpected sequence number indicating that one or more packets are missing or out of order. The receiver software layers will discard packets that do not match the expected sequence number, and the sender software layers will detect that it
- 28. 8 1. INTRODUCTION data link layer data link layer send credits data packets flow ctrl packets Figure 1.4: Example of credit-based flow control across a network link. has not received an acknowledgment packet and will cause a sender timeout which prompts the “send window” — packets sent since the last acknowledgment was received — to be retransmitted. This algorithm is referred to as go-back-N since the sender will “go back” and retransmit the last N (send window) packets. 1.6.2 LOSSLESS FLOW CONTROL Lossless flow control implies that packets are never dropped as a results of lack of buffer space (i.e., in the presence of congestion). Instead, it provides back pressure to indicate the absence of available buffer space in the resource being managed. 1.6.2.1 Stop/Go (XON/XOFF) flow control A common approach is XON/XOFF or stop/go flow control. In this approach, the receiver provides simple handshaking to the sender indicating whether it is safe (XON) to transmit, or not (XOFF). The sender is able to send flits until the receiver asserts stop (XOFF).Then, as the receiver continues to process packets from the input buffer freeing space, and when a threshold is reached the receiver will assert the XON again allowing the sender to again start sending. This Stop/Go functionality correctly manages the resource and avoids overflow as long as the time at which XON is asserted again (i.e., the threshold level in the input buffer) minus the time XOFF is asserted and the buffer is sufficient to allow any in-flight flits to land. This slack in the buffer is necessary to act as a flow control shock absorber for outstanding flits necessary to cover the propagation delay of the flow control signals. 1.6.2.2 Credit-based flow control Credit based flow control (Figure 1.4) provides more efficient use of the buffer resources.The sender maintains a count of the number of available credits, which represent the amount of free space in the receiver’s input buffer. A separate count is used for each virtual channel (VC) [21]. When a new
- 29. 1.7. THE RISE OF ETHERNET 9 packet arrives at the output port, the sender checks the available credit counter. For wormhole flow control [20] across the link, the sender’s available credit needs to only be one or more. For virtual cut-through (VCT) [20, 22] flow control across the link, the sender’s available credit must be more than the size of the packet. In practice, the switch hardware doesn’t have to track the size of the packet in order to allow VCT flow control. The sender can simply check the available credit count is larger than the maximum packet size. 1.7 THE RISE OF ETHERNET It may be an extreme example comparing a typical datacenter server to a state-of-the-art super- computer node, but the fact remains that Ethernet is gaining a significant foothold in the high- performance computing space with nearly 50% of the systems on the TOP500 list [62] using Gi- gabit Ethernet as shown in Figure 1.5(b). Infiniband (includes SDR, DDR and QDR) accounts for 41% of the interconnects leaving very little room for proprietary networks. The landscape was very different in 2002, as shown in Figure 1.5(a), where Myrinet accounted for about one third of the system interconnects. The IBM SP2 interconnect accounted for about 18%, and the remaining 50% of the system interconnects were split among about nine different manufacturers. In 2002, only about 8% of the TOP500 systems used gigabit Ethernet, compared to the nearly 50% in June of 2010. 1.8 SUMMARY Nodoubt“cloudcomputing”benefitedfromthiswildgrowthandacceptanceintheHPCcommunity, driving prices down and making more reliable parts. Moving forward we may see even further consolidation as 40 Gig Ethernet converges with some of the Infiniband semantics with RDMA over Ethernet (ROE). However, a warehouse-scale computer (WSC) [9] and a supercomputer have different usage models. For example, most supercomputer applications expect to run on the machine in a dedicated mode, not having to compete for compute, network, or IO resources with any other applications. Supercomputing applications will commonly checkpoint their dataset, since the MTBF of a large system is usually measured in 10s of hours.Supercomputing applications also typically run with a dedicated system, so QoS demands are not typically a concern. On the other hand, a datacenter will run a wide variety of applications, some user-facing like Internet email, and others behind the scenes. The workloads vary drastically, and programmers must learn that hardware can, and does, fail and the application must be fault-aware and deal with it gracefully. Furthermore, clusters in the datacenter are often shared across dozens of applications,so performance isolation and fault isolation are key to scaling applications to large processor counts. Choosing the “right” topology is important to the overall system performance. We must take into account the flow control, QoS requirements, fault tolerance and resilience, as well as workloads to better understand the latency and bandwidth characteristics of the entire system. For example,
- 30. 10 1. INTRODUCTION (a) 2002 (b) 2010 Figure 1.5: Breakdown of supercomputer interconnects from the Top500 list.
- 31. 1.8. SUMMARY 11 topologies with abundant path diversity are able to find alternate routes between arbitrary endpoints. This is only one aspect of topology choice that we will consider in subsequent chapters.
- 33. 13 C H A P T E R 2 Background Over the past three decades, Moore’s Law has ushered in an era where transistors within a single silicon package are abundant; a trend that system architects took advantage of to create a class of many-core chip multiprocessors (CMPs) which interconnect many small processing cores using an on-chip network. However, the pin density, or number of signal pins per unit of silicon area, has not kept up with this pace. As a result pin bandwidth, the amount of data we can get on and off the chip package, has become a first-order design constraint and precious resource for system designers. 2.1 INTERCONNECTION NETWORKS The components of a computer system often have to communicate to exchange status information, or data that is used for computation. The interconnection network is the substrate over which this communication takes place. Many-core CMPs employ an on-chip network for low-latency, high- bandwidth load/store operations between processing cores and memory,and among processing cores within a chip package. Processor, memory, and its associated IO devices are often packaged together and referred to as a processing node. The system-level interconnection network connects all the processing nodes according to the network topology. In the past, system components shared a bus over which address and data were exchanged, however, this communication model did not scale as the number of components sharing the bus increased. Modern interconnection networks take advantage of high- speed signaling [28] with point-to-point serial links providing high-bandwidth connections between processors and memory in multiprocessors [29, 32], connecting input/output (IO) devices [31, 51], and as switching fabrics for routers. 2.2 TECHNOLOGY TRENDS There are many considerations that go into building a large-scale cluster computer, many of which revolve around its cost effectiveness, in both capital (procurement) cost and operating expense. Al- though many of the components that go into a cluster each have different technology drivers which blurs the line that defines the optimal solution for both performance and cost. This chapter takes a look at a few of the technology drivers and how they pertain to the interconnection network. The interconnection network is the substrate over which processors, memory and I/O devices interoperate. The underlying technology from which the network is built determines the data rate, resiliency,and cost of the network.Ideally,the processor,network,and I/O devices are all orchestrated
- 34. 14 2. BACKGROUND in a way that leads to a cost-effective, high-performance computer system. The system, however, is no better than the components from which it is built. The basic building block of the network is the switch (router) chip that interconnects the processingnodesaccordingtosomeprescribedtopology.Thetopologyandhowthesystemispackaged are closely related; typical packaging schemes are hierarchical – chips are packaged onto printed circuit boards,which in turn are packaged into an enclosure (e.g.,rack),which are connected together to create a single system. ITRS Trend Figure 2.1: Off-chip bandwidth of prior routers, and ITRS predicted growth. The past 20 years has seen several orders of magnitude increase in off-chip bandwidth spanning from several gigabits per second up to several terabits per second today. The bandwidth shown in Figure 2.1 plots the total pin bandwidth of a router – i.e., equivalent to the total number of signals times the signaling rate of each signal – and illustrates an exponential increase in pin bandwidth. Moreover, we expect this trend to continue into the next decade as shown by the International Roadmap for Semiconductors (ITRS) in Figure 2.1, with 1000s of pins per package and more than 100 Tb/s of off-chip bandwidth. Despite this exponential growth, pin and wire density simply does not match the growth rates of transistors as predicted by Moore’s Law.
- 35. 2.2. TECHNOLOGY TRENDS 15 0 10 20 30 40 50 60 70 80 90 100 0.00 0.20 0.40 0.60 0.80 1.00 offered load latency (a) Load versus latency for an ideal M/D/1 queue model. unloaded network latency saturation Average Accepted Bandwidth (Mb/s) Offered Load (Mb/s) Average Message Latency (μs) (b) Measured data showing offered load (Mb/s) versus latency (μs) with average accepted throughput (Mb/s) overlaid to demonstrate saturation in a real network. Figure 2.2: Network latency and bandwidth characteristics.
- 36. 16 2. BACKGROUND 2.3 TOPOLOGY, ROUTING AND FLOW CONTROL Before diving into details of what drives network performance, we pause to lay the ground work for some fundamental terminology and concepts. Network performance is characterized by its latency and bandwidth characteristics as illustrated in Figure 2.2. The queueing delay, Q(λ), is a function of the offered load (λ) and described by the latency-bandwidth characteristics of the network. An approximation of Q(λ) is given by an M/D/1 queue model, Figure 2.2(a). If we overlay the average accepted bandwidth observed by each node, assuming benign traffic, we Figure 2.2(b). Q(λ) = 1 1 − λ (2.1) When there is very low offered load on the network, the Q(λ) delay is negligible. However, as traffic intensity increases, and the network approaches saturation, the queueing delay will dominate the total packet latency. The performance and cost of the interconnect are driven by a number of design factors, including topology,routing,flow control,and message efficiency.The topology describes how network nodes are interconnected and determines the path diversity — the number of distinct paths between any two nodes. The routing algorithm determines which path a packet will take in such as way as to load balance the physical links in the network. Network resources (primarily buffers for packet storage) are managed using a flow control mechanism. In general, flow control happens at the link- layer and possibly end-to-end.Finally,packets carry a data payload and the packet efficiency determines the delivered bandwidth to the application. While recent many-core processors have spurred a 2× and 4× increase in the number of processing cores in each cluster, unless network performance keeps pace, the effects of Amdahl’s Law will become a limitation. The topology, routing, flow control, and message efficiency all have first-order affects on the system performance, thus we will dive into each of these areas in more detail in subsequent chapters. 2.4 COMMUNICATION STACK Layers of abstraction are commonly used in networking to provide fault isolation and device in- dependence. Figure 2.3 shows the communication stack that is largely representative of the lower four layers of the OSI networking model. To reduce software overhead and the resulting end-to- end latency, we want a thin networking stack. Some of the protocol processing that is common in Internet communication protocols is handled in specialized hardware in the network interface controller (NIC). For example, the transport layer provides reliable message delivery to applications and whether the protocol bookkeeping is done in software (e.g.,TCP) or hardware (e.g., Infiniband reliable connection) directly affects the application performance.The network layer provides a logical namespace for endpoints (and possibly switches) in the system. The network layer handles pack- ets, and provides the routing information identifying paths through the network among all source, destination pairs. It is the network layer that asserts routes, either at the source (i.e., source-routed)
- 37. 2.4. COMMUNICATION STACK 17 Network Data Link Physical Transport Network Data Link Physical Transport end-to-end flow control, reliable message delivery routing, node addressing, load balancing link-level flow control, data-link layer reliable delivery physical encoding (e.g. 8b10b) byte and lane alignment, physical media encoding Interconnection Network Figure 2.3: The communication stack. or along each individual hop (i.e., distributed routing) along the path. The data link layer provides link-level flow control to manage the receiver’s input buffer in units of flits (flow control units).The lowest level of the protocol stack, the physical media layer, is where data is encoded and driven onto the medium. The physical encoding must maintain a DC-neutral transmission line and commonly uses 8b10b or 64b66b encoding to balance the transition density. For example, a 10-bit encoded value is used to represent 8-bits of data resulting in a 20% physical encoding overhead. SUMMARY Interconnection networks are a critical component of modern computer systems. The emergence of cloud computing, which provides a homogenous cluster using conventional microprocessors and common Internet communication protocols aimed at providing Internet services (e.g., email, Web search, collaborative Internet applications, streaming video, and so forth) at large scale. While In- ternet services themselves may be insensitive to latency, since they operate on human timescales measured in 100s of milliseconds, the backend applications providing those services may indeed require large amounts of bandwidth (e.g., indexing the Web) and low latency characteristics. The programming model for cloud services is built largely around distributed message passing,commonly implemented around TCP (transport control protocol) as a conduit for making a remote procedure call (RPC). Supercomputing applications, on the other hand, are often communication intensive and can be sensitive to network latency.The programming model may use a combination of shared memory and message passing (e.g., MPI) with often very fine-grained communication and synchronization
- 38. 18 2. BACKGROUND needs. For example, collective operations, such as global sum, are commonplace in supercomputing applications and rare in Internet services. This is largely because Internet applications evolved from simple hardware primitives (e.g.,low-cost ethernet NIC) and common communication models (e.g., TCP sockets) that were incapable of such operations. As processor and memory performance continues to increase, the interconnection network is becoming increasingly important and largely determines the bandwidth and latency of remote memory access. Going forward, the emergence of super datacenters will convolve into exa-scale parallel computers.
- 39. 19 C H A P T E R 3 Topology Basics The network topology — describing precisely how nodes are connected — plays a central role in both the performance and cost of the network. In addition, the topology drives aspects of the switch design (e.g., virtual channel requirements, routing function, etc), fault tolerance, and sensitivity to adversarial traffic. There are subtle yet very practical design issues that only arise at scale; we try to highlight those key points as they appear. 3.1 INTRODUCTION Many scientific problems can be decomposed into a 3-D structure that represents the basic building blocks of the underlying phenomenon being studied. Such problems often have nearest neighbor communication patterns, for example, and lend themselves nicely to k-ary n-cube networks. A high-performance application will often use the system dedicated to provide the necessary perfor- mance isolation, however, a large production datacenter cluster will often run multiple applications simultaneously with varying workloads and often unstructured communication patterns. The choice of topology is largely driven by two factors: technology and packaging constraints. Here, technology refers to the underlying silicon from which the routers are fabricated (i.e., node size, pin density, power, etc) and the signaling technology (e.g., optical versus electrical). The packaging constraints will determine the compute density, or amount of computation per unit of area on the datacenter floor. The packaging constraints will also dictate the data rate (signaling speed) and distance over which we can reliably communicate. As a result of evolving technology,the topologies used in large-scale systems have also changed. Many of the earliest interconnection networks were designed using topologies such as butterflies or hypercubes, based on the simple observation that these topologies minimized hop count. Analysis by both Dally [18] and Agarwal [5] showed that under fixed packaging constraints, a low-radix network offered lower packet latency and thus better performance. Since the mid-1990s, k-ary n-cube networks were used by several high-performance multiprocessors such as the SGI Origin 2000 hypercube [43], the 2-D torus of the Cray X1 [16], the 3-D torus of the Cray T3E [55] and XT3 [12, 17] and the torus of the Alpha 21364 [49] and IBM BlueGene [35]. However, the increasing pin bandwidth has recently motivated the migration towards high-radix topologies such as the radix-64 folded-Clos topology used in the Cray BlackWidow system [56]. In this chapter, we will discuss mesh/torus topologies while in the next chapter, we will present high-radix topologies.
- 40. 20 3. TOPOLOGY BASICS 3.2 TYPES OF NETWORKS Topologies can be broken down into two different genres: direct and indirect [20]. A direct network has processing nodes attached directly to the switching fabric; that is, the switching fabric is dis- tributed among the processing nodes. An indirect network has the endpoint network independent of the endpoints themselves – i.e., dedicated switch nodes exist and packets are forwarded indirectly through these switch nodes. The type of network determines some of the packaging and cabling requirements as well as fault resilience. It also impacts cost, for example, since a direct network can combine the switching fabric and the network interface controller (NIC) functionality in the same silicon package. An indirect network typically has two separate chips, with one for the NIC and another for the switching fabric of the network. Examples of direct network include mesh, torus, and hypercubes discussed in this chapter as well as high-radix topologies such as the flattened butterfly described in the next chapter. Indirect networks include conventional butterfly topology and fat-tree topologies. The term radix and dimension are often used to describe both types of networks but have been used differently for each network. For an indirect network, radix often refers to the number of ports of a switch, and the dimension is related to the number of stages in the network. However, for a direct network, the two terminologies are reversed – radix refers to the number of nodes within a dimension, and the network size can be further increased by adding multiple dimensions. The two terms are actually a duality of each other for the different networks – for example, in order to reduce the network diameter, the radix of an indirect network or the dimension of a direct network can be increased. To be consistent with existing literature, we will use the term radix to refer to different aspects of a direct and an indirect network. 3.3 MESH,TORUS, AND HYPERCUBES The mesh,torus and hypercube networks all belong to the same family of direct networks often referred to as k-ary n-mesh or k-ary n-cube.The scalability of the network is largely determined by the radix, k,and number of dimensions,n,with N = kn total endpoints in the network.In practice,the radix of the network is not necessarily the same for every dimension (Figure 3.2). Therefore, a more general way to express the total number of endpoints is given by Equation 3.1. N = n−1 i=0 ki (3.1) 4 3 2 1 6 5 7 0 4 3 2 1 6 5 7 0 (a) 8-ary 1-mesh. (b) 8-ary 1-cube. Figure 3.1: Mesh (a) and torus (b) networks.
- 41. 3.3. MESH,TORUS, AND HYPERCUBES 21 Mesh and torus networks (Figure 3.1) provide a convenient starting point to discuss topology tradeoffs. Starting with the observation that each router in a k-ary n-mesh, as shown in Figure 3.1(a), requires only three ports; one port connects to its neighboring node to the left, another to its right neighbor, and one port (not shown) connects the router to the processor. Nodes that lie along the edge of a mesh, for example nodes 0 and 7 in Figure 3.1(a), require one less port. The same applies to k-ary n-cube (torus) networks. In general, the number of input and output ports, or radix of each router is given by Equation 3.2. The term “radix” is often used to describe both the number of input and output ports on the router, and the size or number of nodes in each dimension of the network. r = 2n + 1 (3.2) The number of dimensions (n) in a mesh or torus network is limited by practical packaging constraints with typical values of n=2 or n=3. Since n is fixed we vary the radix (k) to increase the size of the network. For example, to scale the network in Figure 3.2a from 32 nodes to 64 nodes, we increase the radix of the y dimension from 4 to 8 as shown in Figure 3.2b. 4 3 2 0 1 6 5 7 12 11 10 8 9 14 13 15 20 19 18 16 17 22 21 23 28 27 26 24 25 30 29 31 4 3 2 0 1 6 5 7 12 11 10 8 9 14 13 15 20 19 18 16 17 22 21 23 28 27 26 24 25 30 29 31 36 35 34 32 33 38 37 39 44 43 42 40 41 46 45 47 52 51 50 48 49 54 53 55 60 59 58 56 57 62 61 63 (a) (8,4)-ary 2-mesh (b) 8-ary 2-mesh. Figure 3.2: Irregular (a) and regular (b) mesh networks. Since a binary hypercube (Figure 3.4) has a fixed radix (k=2), we scale the number of dimen- sions (n) to increase its size. The number of dimensions in a system of size N is simply n = lg2(N) from Equation 3.1. r = n + 1 = lg2(N) + 1 (3.3) As a result, hypercube networks require a router with more ports (Equation 3.3) than a mesh or torus. For example, a 512 node 3-D torus (n=3) requires seven router ports, but a hypercube requires n = lg2(512) + 1 = 10 ports. It is useful to note, an n-dimension binary hypercube is isomorphic to
- 42. 22 3. TOPOLOGY BASICS a n 2 -dimension torus with radix 4 (k=4). Router pin bandwidth is limited, thus building a 10-ported router for a hypercube instead of a 7-ported torus router may not be feasible without making each port narrower. 3.3.1 NODE IDENTIFIERS The nodes in a k-ary n-cube are identified with an n-digit, radix k number. It is common to refer to a node identifier as an endpoint’s “network address.” A packet makes a finite number of hops in each of the n dimensions. A packet may traverse an intermediate router, ci, en route to its destination. When it reaches the correct ordinate of the destination, that is ci = di, we have resolved the ith dimension of the destination address. 3.3.2 k-ARY n-CUBE TRADEOFFS The worst-case distance (measured in hops) that a packet must traverse between any source and any destination is called the diameter of the network. The network diameter is an important metric as it bounds the worst-case latency in the network. Since each hop entails an arbitration stage to choose the appropriate output port, reducing the network diameter will, in general, reduce the variance in observed packet latency. The network diameter is independent of traffic pattern, and is entirely a function of the topology, as shown in Table 3.1 Table 3.1: Network diameter and average latency. Diameter Average Network (hops) (hops) mesh k − 1 (k + 1)/3 torus k/2 k/4 hypercube n n/2 flattened butterfly n + 1 n + 1 − (n − 1)/k from/to 0 1 2 3 4 5 6 7 8 0 0 1 2 3 4 5 6 7 8 1 1 0 1 2 3 4 5 6 7 2 2 1 0 1 2 3 4 5 6 3 3 2 1 0 1 2 3 4 5 4 4 3 2 1 0 1 2 3 4 5 5 4 3 2 1 0 1 2 3 6 6 5 4 3 2 1 0 1 2 7 7 6 5 4 3 2 1 0 1 8 8 7 6 5 4 3 2 1 0 from/to 0 1 2 3 4 5 6 7 8 0 0 1 2 3 4 4 3 2 1 1 1 0 1 2 3 4 4 3 2 2 2 1 0 1 2 3 4 4 3 3 3 2 1 0 1 2 3 4 4 4 4 3 2 1 0 1 2 3 4 5 4 4 3 2 1 0 1 2 3 6 3 4 4 3 2 1 0 1 2 7 2 3 4 4 3 2 1 0 1 8 1 2 3 4 4 3 2 1 0 (a) radix-9 mesh (b) radix-9 torus Figure 3.3: Hops between every source, destination pair in a mesh (a) and torus (b). In a mesh (Figure 3.3), the destination node is, at most, k-1 hops away. To compute the average, we compute the distance from all sources to all destinations, thus a packet from node 1 to
- 43. 3.3. MESH,TORUS, AND HYPERCUBES 23 node 2 is one hop, node 1 to node 3 is two hops, and so on. Summing the number of hops from each source to each destination and dividing by the total number of packets sent k(k-1) to arrive at the average hops taken. A packet traversing a torus network will use the wraparound links to reduce the average hop count and network diameter.The worst-case distance in a torus with radix k is k/2, but the average distance is only half of that, k/4. In practice, when the radix k of a torus is odd, and there are two equidistant paths regardless of the direction (i.e., whether the wraparound link is used) then a routing convention is used to break ties so that half the traffic goes in each direction across the two paths. A binary hypercube (Figure 3.4) has a fixed radix (k=2) and varies the number of dimensions (n) to scale the network size. Each node in the network can be viewed as a binary number, as shown in Figure 3.4. Nodes that differ in only one digit are connected together. More specifically, if two nodes differ in the ith digit, then they are connected in the ith dimension. Minimal routing in a hypercube will require, at most, n hops if the source and destination differ in every dimension, for example, traversing from 000 to 111 in Figure 3.4. On average, however, a packet will take n/2 hops. 010 000 011 001 110 100 111 101 x y z Figure 3.4: A binary hypercube with three dimensions. SUMMARY This chapter provided an overview of direct and indirect networks, focusing on topologies built from low-radix routers with a relatively small number of wide ports. We describe key performance metrics of diameter and average hops and discuss tradeoffs.Technology trends motivated the use of low-radix topologies in the 80s and the early 90s.
- 44. 24 3. TOPOLOGY BASICS In practice, there are other issues that emerge as the system architecture is considered as a whole; such as, QoS requirements, flow control requirements, and tolerance for latency variance. However,these are secondary to the guiding technology (signaling speed) and packaging and cooling constraints.In the next chapter,we describe how evolving technology motivates the use of high-radix routers and how different high-radix topologies can efficiently exploit these many-ported switches.
- 45. 25 C H A P T E R 4 High-Radix Topologies Dally [18] and Agarwal [5] showed that under fixed packaging constraints, lower radix networks offered lower packet latency. As a result, many studies have focused on low-radix topologies such as the k-ary n-cube topology discussed in Chapter 3.The fundamental result of these authors still holds – technology and packaging constraints should drive topology design. However, what has changed in recent years are the topologies that these constraints lead us toward. In this section, we describe the high-radix topologies that can better exploit today’s technology. (a) radix-16 one-dimensional torus with each unidirectional link L lanes wide. (b) radix-4 two-dimensional torus with each unidirectional link L/2 lanes wide. Figure 4.1: Each router node has the same amount of pin bandwidth but differ in the number of ports. 4.1 TOWARDS HIGH-RADIX TOPOLOGIES Technology trends and packaging constraints can and do have a major impact on the chosen topology. For example, consider the diagram of two 16-node networks in Figure 4.1. The radix-16 one- dimensional torus in Figure 4.1a has two ports on each router node; each port consists of an input
- 46. 26 4. HIGH-RADIX TOPOLOGIES and output and are L lanes wide. The amount of pin bandwidth off each router node is 4 × L. If we partitioned the router bandwidth slightly differently, we can make better use of the bandwidth as shown in Figure 4.1b. We transformed the one-dimensional torus of Figure 4.1a into a radix-4 two-dimensional torus in Figure 4.1b, where we have twice as many ports on each router, but each port is only half as wide — so the pin bandwidth on the router is held constant. There are several direct benefits of the high-radix topology in Figure 4.1b compared to its low-radix topology in Figure 4.1a: (a) by increasing the number of ports on each router, but making each port narrower, we doubled the amount of bisection bandwidth, and (b) we decreased the average number of hops by half. The topology in Figure 4.1b requires longer cables which can adversely impact the signaling rate since the maximum bandwidth of an electrical cable drops with increasing cable length since signal attenuation due to skin effect and dielectric absorption increases linearly with distance. 4.2 TECHNOLOGY DRIVERS The trend toward high-radix networks is being driven by several technologies: • high-speed signaling, allowing each channel to be narrower while still providing the same bandwidth, • affordable optical signaling through CMOS photonics and active optical cables that decouple data rate from cable reach, and • new router microarchitectures that scale to high port counts and exploit the abundant wire and transistor density of modern CMOS devices. The first two items are described further in this section while the router microarchitecture details will be discussed in Chapter 6. 4.2.1 PIN BANDWIDTH As described earlier in Chapter 2, the amount of total pin bandwidth has increased at a rate of 100× over each decade for the past 20-25 years. To understand how this increased pin bandwidth affects the optimal network radix, consider the latency (T ) of a packet traveling through a network. Under low loads, this latency is the sum of header latency and serialization latency. The header latency (Th) is the time for the beginning of a packet to traverse the network and is equal to the number of hops (H) a packet takes times a per hop router delay (tr). Since packets are generally wider than the network channels, the body of the packet must be squeezed across the channel, incurring an additional serialization delay (Ts). Thus, total delay can be written as T = Th + Ts = Htr + L/b (4.1)
- 47. 4.2. TECHNOLOGY DRIVERS 27 where L is the length of a packet, and b is the bandwidth of the channels. For an N node network with radix k routers (k input channels and k output channels per router), the number of hops1 must be at least 2logkN. Also, if the total bandwidth of a router is B, that bandwidth is divided among the 2k input and output channels and b = B/2k. Substituting this into the expression for latency from Equation (4.1) T = 2tr logk N + 2kL/B (4.2) Then, setting dT/dk equal to zero and isolating k gives the optimal radix in terms of the network parameters, k log2 k = Btr log N L (4.3) In this differentiation, we assume B and tr are independent of the radix k. Since we are evaluating the optimal radix for a given bandwidth, we can assume B is independent of k. The tr parameter is a function of k but has only a small impact on the total latency and has no impact on the optimal radix. Router delay tr can be expressed as the number of pipeline stages (P) times the cycle time (tcy). As radix increases, the router microarchitecture can be designed where tcy remains constant and P increases logarithmically. The number of pipeline stages P can be further broken down into a component that is independent of the radix X and a component which is dependent on the radix Y log2 k. 2 Thus, router delay (tr) can be rewritten as tr = tcyP = tcy(X + Y log2 k) (4.4) If this relationship is substituted back into Equation (4.2) and differentiated, the dependency on radix k coming from the router delay disappears and does not change the optimal radix. Intuitively, although a single router delay increases with a log(k) dependence, the effect is offset in the network by the fact that the hop count decreases as 1/ log(k) and as a result, the router delay does not significantly affect the optimal radix. In Equation (4.2), we also ignore time of flight for packets to traverse the wires that make up the network channels. The time of flight does not depend on the radix(k) and thus has minimal impact on the optimal radix. Time of flight is D/v where D is the total physical distance traveled by a packet, and v is the propagation velocity. As radix increases, the distance between two router nodes increases. However, the total distance traveled by a packet will be approximately equal since the lower-radix network requires more hops. 3 From Equation (4.3),we refer to the quantity A = Btr log N L as the aspect ratio of the router [42]. This aspect ratio impacts the router radix that minimizes network latency.A high aspect ratio implies a “tall, skinny” router (many, narrow channels) minimizes latency, while a low ratio implies a “short, fat” router (few, wide channels). 1Uniform traffic is assumed and 2logkN hops are required for a non-blocking network. 2For example, routing pipeline stage is often independent of the radix while the switch allocation is dependent on the radix. 3The time of flight is also dependent on the packaging of the system but we ignore packaging in this analysis.
- 48. 28 4. HIGH-RADIX TOPOLOGIES 1996 2003 2010 1991 1 10 100 1000 10 100 1000 10000 Aspect Ratio Optimal Radix (k) Figure 4.2: Relationship between the optimal radix for minimum latency and router aspect ratio. The labeled points show the approximate aspect ratio for a given year’s technology with a packet size of L=128 bits 0 50 100 150 200 250 300 0 50 100 150 200 250 radix latency (nsec) 2003 technology 2010 technology 0 1 2 3 4 5 6 7 8 0 50 100 150 200 250 radix cost ( # of 1000 channels) 2003 technology 2010 technology (a) (b) Figure 4.3: Latency (a) and cost (b) of the network as the radix is increased for two different technologies. A plot of the minimum latency radix versus aspect ratio is shown in Figure 4.2 annotated with aspect ratios from several years.These particular numbers are representative of large supercomputers with single-word network accesses4, but the general trend of the radix increasing significantly over time remains. Figure 4.3(a) shows how latency varies with radix for 2003 and 2010 aspect ratios. As radix is increased, latency first decreases as hop count, and hence Th, is reduced. However, beyond a certain radix, serialization latency begins to dominate the overall latency and latency increases. As bandwidth, and hence aspect ratio, is increased, the radix that gives minimum latency also increases. For 2004 technology (aspect ratio = 652), the optimum radix is 45 while for 2010 technology (aspect ratio = 3013) the optimum radix is 128. 4The 1996 data is from the Cray T3E [55] (B=48Gb/s, tr =40ns, N=2048), the 2003 data is combined from the Alpha 21364 [49] and Velio VC2002 [20] (1Tb/s, 10ns, 4096), and the 2010 data was estimated as (20Tb/s, 2ns, 8192).
- 49. Discovering Diverse Content Through Random Scribd Documents
- 50. any earlier point it could have been treated as referring to mind only by anticipation. Here, however, the problem can no longer be deferred. The “free mind” does not explain itself and cannot stand alone. Its impulses cannot be ordered, or, in other words, its purposes cannot be made determinate, except in an actual system of selves. Except by expressing itself in relation to an ordered life, which implies others, it cannot exist. And, therefore, not something additional and parallel to it, which might or might not exist, but a necessary form of its own action as real and determinate, is the actual fabric in which it utters itself as Society and the State. This is what Hegel treats in the second division of the Philosophy of Mind under the name of Mind Objective. It is not for him ultimate. A particular society stands in time, and is open to criticism and to destruction. Beyond it lies the reality, continuous with mind as known in the State, but eternal as the former is perishable, which as Absolute Mind is open to human experience in Art, Religion, and Philosophy. We will pursue in the following chapter Hegel’s analysis of the modern State as Mind Objective, a magnified edition, so to speak, of Plato’s Republic, bringing before the eye in full detail distinctions and articulations which were there invisible. {256} CHAPTER X. THE ANALYSIS OF A MODERN STATE. HEGEL’S “PHILOSOPHY OF RIGHT.”
- 51. 1. We are about to analyse a modern State into groups of facts which are also ways of thinking. And a question may arise in what sense the connection is to be understood which will be alleged to bind together these groups of facts or points of view. When it is urged that group b or view b is suggested and made necessary by the shortcomings of group a or view a, does this imply that group a or its idea came into existence first, and group b or the notion of it sprang up subsequently or as an effect of the former? And could such a relation be reasonably maintained as between the component parts of a unity like the State? An answer may be indicated as follows. We are dealing, in society and in the State, with an ideal fact. As a fact, a form of life, society has always been a many-sided creature, meeting the varied needs of human nature by functions no less varied. As an ideal fact, however, its advance has partaken of the nature of theoretical progress. In the continuous attempt to deal satisfactorily {257} with the needs of intelligent beings, the mind, the intelligent will, has thrown itself with predominant interest now into one of its functions and now into another. And this has not been a chance order of march. Obviously, what it has emphasised and modified in the second place has depended both positively and negatively on what it had emphasised and modified in the first place. Positively, because when one step is thoroughly secured the next may be definitely attempted. Negatively, because the definite attainment of one step exposes the limitations of what has been achieved, and the need for another. At every stage the will is dissatisfied with the expression of itself which it has created. Till some public order has been established, morality can hardly find expression; but when a legal system is thoroughly in force it becomes apparent how far the letter may fall short of the spirit. We see the same action of intelligence in pure theory. Every
- 52. conquest of science leads to a new departure. It suggests it by its success, and demands it by its failure. Now, in science it may or may not be the case that the connection which has led to a discovery enters permanently as a discernible factor into the structure of knowledge. The re-organisation of experience may sweep away the steps which led to it. But in the living fact of society this is not so. Its many sides are actual and persist, and the emphasis laid from time to time on the principle of each—e.g. on positive law, on family ties, on economic bonds— merely serves to accent an element which has its permanent place in the whole. Thus, there must always be family ties and economic bonds. But at one time everything tends to be construed {258} in terms of kinship, at another time in terms of exchange. And the tendency means a difference of actual balance between the functions as well as a different theory. The positive and negative connection of elements like these, the true place and limit of each, is permanently rooted in human nature, but may be elucidated by the explicit logic of their attempt and failure to give the tone to the whole social fabric. It follows that the social whole grows, like a great theory, in adequacy to the needs which are its facts; and the dissatisfaction of the will with its own expression, in other words, the contradictions which practical intelligence is continually attempting to remove, becomes more like suggestion than flat contradiction—or change, as we say, becomes less revolutionary. It may seem to be a difference between the social whole and a scientific theory that the former, as it grows, creates new difficulties, by creating new and freshly contradictory matter, as in the social problems of civilisation; while the latter, as we imagine, deals with an unchanging experience. But this distinction is less true than it appears, and the
- 53. comparison with the growth of a theory will always throw light on the true nature of the will and its continuous effort to satisfy itself. 2. Right or Law may be taken in the widest sense as including the whole manifestation of Will in an actual world—“the actual body of all the conditions of freedom,” [1] “the realm of realised freedom, the world of mind produced out of itself, as a second nature.” [2] It is a merit of the German {259} term “Recht” [3] that it maintains the connection between the law and the spirit of law, [4] and almost of itself prohibits the separation between positive law, and will, custom or sentiment, which underlies such a theory as Austin’s. [1] Hegel, Philos. of Mind (E. Tr.), p. 104. Cf. defs. quoted from Green, p. 203 above. [2] Rechtsphil., sect. 4. [3] Cf. the Greek’s idea of “nomos.” [4] See ch. ii. above on Montesquieu and Rousseau. This whole sphere of Right or Law, the mind as actualised in Society and the State, naturally divides itself on the principle which has just been explained, into three connected groups of ideal facts or points of view. The first, or simplest and most inevitable, of these, may be called the “letter of the law” as we come upon it most especially in the law of property—Shylock’s law—the sheer fact, as it seems, that the world is appropriated by legal “persons.” The second, obviously conditioned by the first both positively and negatively, may be described as the morality of conscience; the revolt of the will against the letter of the law, though this was its
- 54. own direct expression of itself (e.g. in taking things as property); and its demand to recognise as right nothing but what springs from itself as the good will.
- 55. And thirdly, there is the reality or concrete experience in which the two former sets of facts, or ideas, find their true place and justification—the completed theory, so to speak, which adjusts and explains the narrower views founded on one-sided contact with life. This is indicated to consist in “social observance,” or “ethical use and wont”; the system of working mind where the true will appears as incarnate in a way of living. This, {260} like the others, it must be remembered, is a fact, though akin to a theory. Not only does it explain and justify the other factors, but its existence has enabled them to exist, as theirs has also been essential to it. And yet each of the three, as one aspect of society which under certain influences may catch the eye, has at times claimed—is, indeed, constantly claiming—predominance, and has thus brought into relief its own defects and the need of the complementary ideas. We will speak of these moods of mind or kinds of experience in their order, expecting a further sub-division when we come to treat of the third. 3. “Law,” then, in the directest possible sense—the minimum sense, so to speak—is the hard literal fact that it is a rule of the world we live in for things to be appropriated by persons. This is the first or minimum change of the world from mere matter into the instruments of mind, and it is a necessary change. Things have no will of their own, and it is by having a will asserted upon them that they become organs of life. In the same way, it is by assertion in external things that the will first becomes a fact in the material world. Property is “the first reality of freedom.” [1] It is not the mere provision for wants, but the material counterpart of will. Contract belongs to this sphere, the sphere of property. It is an agreement of persons about an external thing—a “common will,” but not one
- 56. “general” or “universal” in its own nature like that involved in the State. [1] Rechtsphil., sect. 41. Not, in its developed form, the first in time. Hegel lays stress on the fact that true, free, property was hardly realised even in his own day. {261} Thus, it is a confusion of spheres to apply the idea of contract to the State, for the State is an imperative necessity of man’s nature as rational, while contract is a mere agreement of certain free persons about certain external things. The idea of the social contract is a confusion of the same type as that by which public rights and functions were treated as private property in the middle age. The attributes of private property are nothing more than the conditions of “personal” existence, and absurdity results if they are transferred to functions of the State. This phase or view of law as, in its letter, an ultimate and absolute rule, may be illustrated, Hegel says, by the Stoic notion that there is only one virtue and one vice; by the Draconic conception that every offence demands the extreme penalty; and by “the barbarity of the formal code of honour, which found in every injury an unpardonable insult.” It might also be illustrated by Austin’s theory of law as a command enforced by a penalty; or by the theories which account for property simply by the fact of occupancy or of labour mixed with the thing. The common point of all these views is that they treat the law, not as a part of a living system, [1] ultimately resting on the will to maintain a certain type of life, but as something absolute in its separateness, and equally sacred in all its accidents and inequalities. [1] See e.g. above, p. 232, how the idea of a system of rights may modify punishment.
- 57. Now, this emphasis and idea of law, being the exaggeration of a single and direct necessity, the {262} necessity of order and property, may be called “primitive” or barbarous, but it cannot of course be identified with the earliest state of social authority known to history or to anthropology. There we should probably find law undifferentiated from custom and from religious sentiment, and consequently, though rigid enough, not in any such one-sided absoluteness as we have been describing. All we can say is that this is the way in which law must come to be regarded whenever its living spirit is forgotten, and an unreal absoluteness is assigned to it; and this connection of principle verifies itself as a fact in recurrent historical phenomena, and in fallacies which perpetually reappear. 4. Within the whole fabric of right or realised will, the element which naturally asserts itself by antagonism to the letter of the law is the morality of conscience, conscientiousness, or the idea of the Good Will. It is connected with the letter of the law, as Hegel puts it, by the various degrees of wrong. The will, that is to say, finds itself at variance in or with [1] the order of law and property which it has created as its direct and necessary step to freedom. Its realised theory, so to speak, is found to break down at a certain point, by being in contradiction with the needs which it was created to meet. “Summum jus, summa injuria”. We may object that the anti-legal will is simply wrong. This may be so, and again it may not be so. What the will has awakened to, whether right or wrong, is {263} that it can acquiesce in nothing which does not come home to it as fulfilling its own principle. What so comes home to it is what it calls “good,” and it cannot accept any order or necessity which it cannot will as good, i.e. as satisfying its own idea.
- 58. [1] “In it,” when my will does not conflict with right as such, but claims the right in an object A to be mine and not yours—a civil dispute. “With it,” when my will rebels, and by its act, so far as in it lies, denies and destroys the whole fabric of right, e.g. takes the object A, without alleging a right to it—theft, a criminal offence, cf. p. 230. When this phase of reaction is pushed to its logical extreme, we have the modern doctrine of my conscience and my pure will. It is the conflict of the inner self with the outer world, expressed in history through the Stoic and through some forms of the Christian consciousness (especially the Protestant consciousness), and in philosophy through the Kantian doctrine of the good will, uttered in the famous sentence, “Nothing can possibly be conceived in the world or out of it which can be called good without qualification except a good will.” [1] Nothing is worth doing but what one ought, and because one ought. [1] Kant, Grundlegung zur Metaphysik d. Sitten, sect. I. The criticism to which this principle has been subjected is familiar to students of ethics. Its point is, in brief, that there is no way of connecting any particular action with the mere idea of a pure will. The forms assumed by evasions of this difficulty, which we fall into when we desire wholly to separate the inner from the outer, or the “ought” from the “is,” are treated by Hegel with unsurpassable vigour and subtlety, as indeed the annihilating criticism of this conception is primarily due to him. The essence of the matter is that the pure will directed towards good for the sake of good, having no real connection with any detailed conduct, may be alleged by self- deception in support of any behaviour whatever, and out of this may
- 59. spring the {264} whole sophistry and hypocrisy of “pure intention.” He makes the shrewd observation, [1] which is still of interest, that the extreme Protestant doctrine of conscience may take the form of ethical vacuity or instability, and that this had in his time been the cause of many Protestants going over to Rome, to secure some sort of moorings, if not precisely the stability of thought. [1] Rechtsphil., sect. 141. Still, out of all this one-sidedness, there survives the permanent necessity that an intelligent being can acquiesce only in what enters into the object of his will. It is his will which affirms the aim to which his nature draws him, and he is absolutely debarred from reposing in anything which does not appeal to his will. The subjective will is the only soil on which freedom can be a reality. So, within the general organism of Right or realised Free-will, we have found two opposite groups of facts—for the aspirations of intelligent beings are facts—or tendencies or theories, which are connected by opposition, and yet are necessary to the expression of the same underlying need—the letter of the law, and the freedom of conscience. 5. Hegel’s name for the third term, which, as he puts it, expresses the “truth” of these extremes, may be rendered “the Ethical System,” or “the Moral Life,” or “Social Ethics.” It expresses “the truth” of the extremes, as a good theory may express the truth of two one-sided views. Only, as we have said, it is a fact as well as a theory, and therefore is something which actually contains what these two views demand, and does the work which they, and the facts they rely on, {265} exhibit as necessary to be done. This relation is not obscure or unprecedented. Every institution, every life, works as a theory,
- 60. and either masters its facts or fails to master them; though not every theory is a life or an institution. The German word which the above-mentioned phrases attempt to render is “Sittlichkeit” The word takes its meaning from “Sitte” which in common usage is equivalent to “custom.” Hegel’s use of the term, in his later writings, as opposed to “Moralität” and as indicating, in comparison with it, a fuller and truer phase of life, is an intentional declaration of war against the Kantian principle of the pure good will, and is the gist of Hegel’s ethico-political view in a nutshell. The word would most naturally apply to the life of a community in which law, custom, and sentiment were not yet very sharply distinguished. According to accepted views, the communities of ancient Greece, before they were stirred by the reflective movement which is associated with the names of Socrates and the Sophists, would be examples of a disposition and order of life which the word “Sittlichkeit” might denote. And it was in the Greek communities, as is shown by the work which he sketched as early as 1802, [1] that Hegel found this suggestion of a whole in which law and custom, duty and disposition, were absolutely at one. He subsequently modified the conception in accordance with the modern idea of freedom, by allowing a greater emphasis and relief to its {266} component parts, and insisting (against Plato’s Republic for instance,) on the principle of individual choice, initiative, and property, as necessary to the complete communion of intelligent beings. As we have just seen, indeed, he introduces reflective morality or conscientiousness into the sphere of Right, to represent the full nature of mind, which is only exhibited in a consciousness which pursues its aims of its own choice and for their own sake.
- 61. [1] The System d. Sittlichkeit. The Rechtsphil. was not published till 1817, in its earliest form. See Wallace, Hegel’s Philosophy of Mind, p. 187. The Ethical System, then, or Social Ethics, is put forward as the ideal fact which includes, and does the work of both the literal law and the moral will, alike in practice and as a theory. It is the idea of freedom developed (i.) into a present world, and (ii.) into the nature of self-consciousness. For (i.), in the first place, the ethical system, or the ways of acting which make up social ethics, constitute a present and actual world. So far it partakes of the nature of the literal law and order, the system of property-holding, which, as we have seen, is all but a natural fact. Social Ethic, we might say, is a physical fact. The bodily habits and external actions of a people incorporate it. It transforms the face of a country, “domesticating the untamed earth.” [1] Each individual has his own bodily existence in a determinate mode as a part of the ethical life of society. The rules and traditions of ethical living are, as has been said, “the nature of things.” They are as hard, as “objective” an order as “sun, moon, mountains, rivers, and all objects of nature.” [2] Man lives according to them before he knows that he {267} does so, and always, in a great degree, independently of knowing that he does so. As this group of facts, or considered from this point of view, the ethical system is the body of the moral world. [1] Aeschylus, Eumenides, I. 14. [2] Rechtsphil., sect. 146. (ii.) But it is also and no less the very nature of self-consciousness. It is as much a demand of man’s intelligence or an inner and universal law as the “pure will” itself. [1] The difference between
- 62. them is that the Ethical System is a system, a world, though from the point of view of will regarded as inner, that is to say, as something which is the motive and fulfills the demand of consciousness. Thus, it bears the character of a thoroughly systematised theory, as contrasted with the idea of a good will, which is a mere general point of view. And it is because of this systematic character that it is enabled to connect the individual or particular will with the universal spirit of the community. It is only in a system that a particular fact can be connected with a universal law, as the planetary motions are with the law of gravitation. The particular will, as we have explained above, is universalised by its relation to a systematic purpose which it partly implies and partly realises. A man wishes for this thing or that thing, but not at any price. The reservations to which his wish is subject, by reason of other purposes and postulates of life, are known to him only in part; but if they could be stated in full, they would constitute the system of his life as realised in the universal life of the community. It is precisely {268} analogous to the process which a common judgment of perception has to go through in becoming a scientific truth—the implications have to be stated in full, and the perception modified in accordance with them. And when this is done, we have no longer a fact, but a science. [1] On all this portion of the subject, see Mr. Bradley’s Essay, “My Station and its Duties,” in Ethical Studies. Regarded from this point of view, as the substance of the individual Will, the Ethical System is the Soul of the moral world.
- 63. In analysing the Ethical System, we shall say nothing of “duties” or “virtues.” Duty is in each case what the relation requires—the attachment of the universal system of will to the individual life. Virtue is a habit of such action, considered as embodied in the nature of an individual. The idea of virtue and virtuousness is not, in Hegel’s view, altogether suitable to the members of an ethical commonwealth. It belongs rather to a time of undeveloped social life, when ethical principles and the realisation of them are ascribed to the nature of peculiarly gifted individuals. Virtue or excellence, to the Greek moralist, for instance, suggested doing something better than the average, or being in some way specially gifted, and it is still apt to indicate the desire to be some thing exceptional, and not simply to find yourself in genuine service. The meaning of the words to-day tends to narrow itself to certain special relations, and does not indicate that life of the member in the whole, which is the essence of what we really value. The Ethical System, or the Order of Social Ethics, then—the mind and conduct of the citizen in Christendom—may be regarded as affirming freedom {269} in three principal aspects, necessarily connected, and supplementing one another. Outwardly these aspects are different groups of facts—different institutions; inwardly they are different moods or dispositions of the one and indivisible human mind. Thus, Hegel’s analysis regards the social whole or system of social ethics from three points of view. First, in respect of the Family; secondly, in respect of what he has entitled Bourgeois Society; and thirdly, in respect of the Political Organism, or the State in the strict sense.
- 64. It is to be borne in mind that, like the three principal divisions in the sphere of Right, these headings represent explicit theories of society, as well as groups of facts. 6. Beginning once more, within an ordered social sphere, at the ethical factor which stands nearest to the natural world, and has taken, so to speak, the minimum step into the realm of purpose and consciousness, we start from the family. As the family exists in a modern civilised community, it is something necessary to society and the State, but absolutely distinct from both. It first (a) represents the fact of the natural basis of social relations, being the embodiment of natural feeling in the form of love, both as between the parents, and as embodied for them in the children. It is in accordance with Hegel’s general views of the meaning of a system that he sees this element of mind primarily represented by the family, as an organ preserved and differentiated ad hoc, and not, or not merely, distributed indefinitely throughout the community. Thus, the modern family represents for him a higher stage {270} of civilisation—an organ to a fuller embodiment of mind —than the clan or tribe, or, in short, than any form of community in which the whole bond of union rests on merely natural feeling, kindness, generosity, or affection. In the nation, indeed, a tinge of natural affection, a colouring of unity by kinship, survives, just as feeling runs through the experience of the individual mind. But the distinctive character of the State is clear intelligence, explicit law and system, and so the natural basis of feeling, though necessary to be preserved and spiritualised, achieves these needs in the family as a special organ, and not in the State as such.
- 65. All those theories, therefore, which tend to assimilate the State to a family by a sort of levelling down of the former or levelling up of the latter (Plato’s Republic, the phalanstery, paternal government, and the like) involve for Hegel a mere confusion of relations. They recognise an element which is essential to society, and may truly be said to be even its foundation. But they do not see its right place in the whole, and do not understand that in order to attain a stronger and deeper unity (which is, in short, a stronger and deeper mind) the different elements must be allowed a greater emphasis and relief, and their respective characteristics must not be slurred or scamped. But (b) in the second place, the family is a factor in the rational whole, the State, though its function par excellence is that of the natural basis of society. Hence its nature and sanction is ethical—it rests neither on mere feeling on {271} the one hand, nor on mere contract on the other. It has a public side, and the acceptance of a universal obligation by a declaration in explicit language (language being the stamp of the universal), in face of the community, is an essential part of marriage, and not a mere accident or accessory, as the votaries of feeling have urged. This view is aimed against the confusion which finds the sole essence of marriage in feeling. This is a perpetually recurring contention, represented in Hegel’s day by Friedrich von Schlegel’s Lucinde, which argues that the form of marriage destroys the value of passion. Hegel’s analyses are everywhere directed against this inability to grasp the distinct sides of a many-sided fact. (c) The ethical aspect of the family [1] shows itself in the nature and organisation of the household, as an institution embodying permanent interests and relations of the two persons who are its
- 66. head, and as an organ of public duties in the bodily and spiritual nurture of the children. The permanent and equal relation of the heads of the household, involved in its nature as the ethical aspect of the family, implies monogamy, and it is the monogamous family alone which can count as a true element of the ethical order. [1] Cf. Green’s Principles of Political Obligation, p. 235. (d) The household, being the true and operative ethical organ which makes parentage into family, is the unit which demands to be respected and protected by the State against the less differentiated forms of consanguinity, such as the clan. The true family starts from marriage and the foundation of a household, and in the early {272} development of law we find the State, with a just instinct, protecting the household against the clan, e.g. by conferring the power of bequest. This power, though now it may imply a discretion mainly hostile to the family, presented itself in early law rather as a means of perpetuating the separate household as against the pretensions of the clan to interfere with its property. Thus, the monogamous family is naturally and necessarily, to some extent, a unit in respect of property; the children, at least, being inevitably under tutelage and incapable of self-support, even if economic equality asserts itself as between husband and wife. This peculiar relation in respect of property is rooted in the unique nature of the household, as an organ for the guardianship of immature lives, and as a unity of feeling rather than of explicit thought. It is noticeable that progress tends to introduce the distinctions of property within its unity [1] (though for children this can never go very far), and very slightly to introduce the relations of the family into the outside world. In as far as such distinctions come to be
- 67. made, the nature and functions of the household being undisturbed, a somewhat higher intensity of ethical union is rendered necessary, and will no doubt assert itself. [1] Married Women’s Property, Protection of Earnings of Children, Property assigned by understanding within household to young children. 7. When the man (or woman [1]) arrives at maturity and leaves the safe harbour of the family, he finds himself, prima facie, isolated in a world {273} of conflicting self-interests. He has his living to make, or his property to administer. He is tied to others, in appearance, only by the system of wants and work, with the elementary function which is necessary to it, viz. its police functions and the administration of justice. [1] Hegel would say only or chiefly the man, who is for him the natural earner and chief of the household. It is this phase of social life, and the temper or disposition corresponding to it, which Hegel indicates by the expression Bourgeois Society. [1] It presents itself to him as the opposite extreme of life and mind to that embodied in the family. It is an aggregate of families—for the units of the Bourgeois Society are heads of households—as seen from the outside, in the great system of industry and business, where a man has to find his work and do it. It is, in mind, the presence of definite though limited aims, calculation and self-interest. [2] [1] Bürgerliche Gesellschaft. “Society,” Wallace points out, is here opposed to “community,” and indicates a looser phase of union.
- 68. [2] Cf. the merchant in Wilhelm Meister’s Lehrjahre, viii. 2. “I can assure you that I never reflected on the State in my life. My tolls, charges, and dues I have paid for no other reason than that it was established usage” (cited from Wallace, Hegel’s Philosophy of Mind, p. cci.). Bourgeois Society is the aspect of the social whole insisted on by the classical political economy, by which, as an achievement in the way of reducing complex appearances to principles, Hegel was much impressed. It is, again, the view of society embodied in the conception of the purely police State, and its principle is confused with that of the State proper by one set of theorists, as that of the family is by another. It is the peculiarity of Hegel’s view—probably {274} the most definitely original, as it is the most famous, of all his political ideas— to contend that this aspect of society, with the form of consciousness belonging to it, is necessary to a modern State. According to his logic, indeed, it is inevitable that every true whole shall have an aspect of “difference,” of breaking up into particulars. The principle of the ancient State, as concentratedly expressed in Plato’s Republic, was weak and undeveloped, and fell short of the true claims of intelligence, [1] just because it dared not really let the individual go—let him assert himself as himself. “Subjectivity” was a principle fatal to it. Not that there was an iron oppression in the States of antiquity. The individual was, for an onlooker, magnificently developed. His limitations were in him, and did not oppress him; but for all that, free choice and the career open to talents were not for him.
- 69. [1] “Was not ideal enough” (Hegel, Geschichte der Philosophie, ii. 254). The “notion” for him necessarily involves identity, differentiation, and re-integration; and in this respect the ancient State falls short of a true notion, while the modern realises it. The modern demand—such is Hegel’s conception—is harder and higher. The individual’s life is not predetermined by his birth, but he is thrown face to face with economic necessity, which is a form of the universal end. He has to strip off his crudeness and vanity, and, of himself, mould himself into something which fulfils a want. This is a step without which there can be no true freedom—the giving one’s self by one’s own act a definite place in the region of external necessity, the “becoming something” or attaching oneself to {275} a definite class of service renderers. Thus, we are startled to find culture or education treated in general, and in respect of its indispensableness, under the head of the Bourgeois Society. For culture is the liberation from one’s own caprices, and the acceptance of a universal task. It is a severe process, and therefore unpopular, but it is a necessary one if we are to have true freedom. The criticism that such a world and temper is the world and temper of self-interest does not appeal strongly to Hegel. We shall have to treat of it more fully below. [1] [1] See p. 291. It may be noted in passing that the insecurity of life, which may seem to attach to dependence on the vast system of wants and work, is more and more seen, as modern economic relations develop, not to be insecurity at all, except in as far as “culture” in the form of industrial training is absent. There is, indeed, in modern life, nowhere any absolute and oyster-like stability. The highest
- 70. stability to be anywhere attained is that due to fitness for service in the interdependent system of needs. [1] [1] I may refer to The Standard of Life, by H. Bosanquet, essay on “Klassenkampf”. Therefore, as Hegel saw, but in more ways than he saw, the system of Bourgeois Society—the economic and industrial world—is not a separate reality, but only an appearance within a larger system. The member of it is not so detached as he may seem, or think. He is within, and sustained by, the general life of the State, as the aims which are his motives in “business” or industry are within {276} and inseparable from the whole structure of his intelligence. Thus, the world of Bourgeois Society—a world, on the whole, of cash nexus and mere protection by the State—has a structure or tendency of its own which brings it back by necessary steps to connection with the State proper or explicit and determinate social unity. It is, we must observe, posterior to the State in time. It is only within the State proper, and resting on its solid power, that such a world as that of Bourgeois Society could arise or be conceivable. Its priority to the State is, like that of the family, the priority of comparative narrowness or simplicity, of dealing with fewer factors, and of representing human nature in a more special, though necessary, aspect. And for this very reason it could not exist by itself. It has not the many-sided vitality indispensable to anything which is to hold its own in the actual world. The working of the Bourgeois Society, then, exhibits an inevitable connection with the State proper, and, so to speak, leads up to it.
- 71. In the first place, the economic world implies the administration of justice. In this, as involving a developed system of civilised law, there is an advance on the “letter of the law” in its crudest and most barbarous acceptation. The system of law of a modern State is, and still more ought to be, [1] a fairly reasonable and intelligible definition of the rights and relations of persons. By this determination the economic system of particular wants and services enters upon a first {277} approximation, as it were, to a unity of principle. The law only professes, indeed, to protect property and exchange, but in doing so it unavoidably recognises that the particular want has a general bearing; for the developed system of law only comes into existence to enable wants to be supplied, and takes its definite shape according to the system of wants. We may illustrate this first approximation to universality, which law confers upon the particulars of private interest, by a suggestive view which M. Durkheim has propounded. [2] He has pointed out that the current formula for social change, “from status to contract,” has a subtler significance than is apt to be recognised. For contract is not really indeterminate, as if it arose in vacuo without a precedent. It runs in forms determined by social experience through law and custom; and thus the law, which professedly aims at protecting property and exchange, necessarily regulates them by the modes in which it chooses to protect them. [1] Hegel pleads strongly for codification. [2] De la Division du Travail Social, 225 ff. A more intimate relation to the State proper—to a definite principle, as we might say, of common good—grows out of the interests of Bourgeois Society which take the shape of what a
- 72. German calls “Police and Corporation,” i.e. State regulation and Trade Societies. The basis of State regulation is the emergence of aspects of common interest in the system of particular interests. The region of particular interests (supply and demand) has an accidental side, and the State has a right and a duty to protect the general good against accidental {278} hindrances. On the whole, no doubt, the right relation between producer and consumer arises of itself, but miscarriages may occur which call for interference on behalf of the explicit [1] principle of the general good. The general possibility of the individual’s obtaining what he wants is a public interest, and the State has a right to intervene with this end in view, both by execution of necessary public works, by sanitary inspection and the like, and by inspection and control of fraud in the case of necessary commodities offered for sale to the general public. For the public offer of goods in daily use is not a purely private concern, but a matter of the general interest. If indeed there was complete official regulation, there would be a risk of getting work like the Pyramids, that represented no private want at all; but yet, in the system of private wants, there is a public interest that demands vigilance. [1] The explicit idea of common good always belongs in Hegel to the State proper. A similar approximation of Bourgeois Society to the State is constituted by the “Corporation,” which rests on the facts of class. Every member of the Bourgeois Society belongs by his vocation to a class, and this breaking up into classes is a consequence of the division of labour which prevails in the economic sphere, disguising the common good as private interest or necessity. But in the
- 73. formation of classes society begins as it were to recover from the dispersion which private interest has occasioned. As a member of his class [1] or {279} “estate,” the citizen acquires solidarity with his fellows, and his particular interest becomes ipso facto a common one. As a member of the class, again, he is, or ought to be, a member of his “trade society” or “corporation.” In this he finds his honour or recognition, [2] a definite standard of life (apart from which he is apt to assert himself by aimless extravagance, for want of a recognised respectability), a standard of work, insurance against misfortune, and (as a candidate for admission) the means of technical education. [1] The term “Stände” it must be remembered, has for a German the association of elements of the representative assembly; “états”, estates of the realm. [2] Cf. the English workman’s phrase, “a good tradesman,” i.e. a competent member of his trade. If the family is the first basis of the State, the classes or estates are the second. The Corporation or Trade Society is a second family to its members. It is the very root of ethical connection between the private and the general [1] interest, and the State should see to it that this root holds as strongly as possible. [2] [1] “We can only say that these men, if they leave us, will bitterly regret it. … The man who is so unselfish as to care nothing for himself or his fellow-men will soon find himself, as years creep over him, and grey hairs and glasses, completely cut out.”—“Branch Trade Report (Birmingham) to National Union of Boot and Shoe Operatives, January, 1896.”
- 74. [2] Sects. 201 and 255. I omit Hegel’s characterisation of the classes, which has a good deal in common with theories which represent occupations as determining character. The contrast between agricultural and industrial or commercial life, between country and town, is of great importance in his view. He almost seems to confine Bourgeois industrialism as such to the life of town- dwellers; though, again, ultimately the whole division into classes is characteristic of Bourgeois Society (cf. sects. 256 and 305). “If,” Hegel writes, [1] “in recent days the “Corporation has been abolished, this has the significance {280} that the individual ought to provide for himself. This may be admitted; but the corporation did not alter the individual’s obligation to earn his livelihood. In our modern States the citizens have only a limited share in the universal business of the State; but it is necessary to permit the ethical human being a universal activity over and above his private end. This universal, which the modern State does not always provide for him, he finds in the Corporation. We saw before that the individual providing for himself in the Bourgeois Society also acts for others. But this unconscious necessity is not enough; it needs the Corporation to bring it to a conscious and thoughtful social ethics. Of course the Corporation needs the higher superintendence of the State, or it would ossify, shrink into its shell, and be degraded into a wretched guild. But in and for itself the Corporation is no closed guild; it is rather the bringing of an isolated trade into an ethical connection, and its admission into a sphere in which it wins strength and honour.” [2] [1] Sect. 255.
- 75. [2] It is obvious that this treatment of associations arising among classes in industry and commerce does not apply in principle exclusively to trade or professional societies. It would include, e.g., Friendly Societies and Co-operative Societies, by which members of the economic world bind themselves together for help, recognition, and the assertion of their general interests. 8. The State proper, or political constitution, presents itself to Hegel as the system in which the family and the Bourgeois Society find their completion and their security. He was early impressed, as we have seen, with the beautiful unity of the ancient Greek commonwealths. And the first and last idea which governs his representations of the modern State is that of the Greek commonwealth enlarged as it was from a sun to a solar {281} system. The family feeling and the individual interest are in the modern State let go, accented, intensified to their uttermost power; and it is out of and because of this immense orbit of its elements that the modern State has its “enormous strength and depth.” It is the typical mind, the very essence of reason, whose completeness is directly as the completeness of each of its terms or sides or factors; and secure in the logical confidence that feeling and self- consciousness, the more they attain their fulness, must return the more certainly to their place in the reasonable system which is their very nature. As ultimate power, the State maintains on one side the attitude of an external necessity towards the spheres of private life, of the family, and of the economic world. It may intervene by force to remove hindrances in the path of the common good, which accident and immaturity may have placed there. But, in its essence, the State is the indwelling and explicit end of these modes of living, and is strong in its union of the universal purpose with the particular interests of mankind. It is, in short, the incarnation of the general or
- 76. Real Will. It has the ethical habit and temper of the family as a pervading basis, combined with the explicit consciousness and purpose of the business world. In the organism of the State, i.e. in as far as we feel and think as citizens, feeling becomes affectionate loyalty, and explicit consciousness becomes political insight. As citizens we both feel and see that the State includes and secures the objects of our affections and our interests; not as separate items, thrown together by chance, but as purposes transformed by their relation to the common good, into {282} which, as we are more or less aware, they necessarily pass. This feeling and insight are the true essence of patriotism. It is easier to be magnanimous than to be merely right, and people prefer to think of patriotism as a readiness to make great sacrifices which are never demanded. But true patriotism is the every-day habit of looking on the commonwealth as our substantive purpose and the foundation of our lives. The division of functions in the State is a necessary condition of its rational organisation. But, as Rousseau had insisted, it is altogether false to regard these separate functions as independent, or as checks on one another. There could be no living unity, if the functions of the State were ultimately independent and negative towards each other. Their differentiation is simply the rational division of labour. The State is an image of a rational conception; it is “a hieroglyph of reason.” Sovereignty, therefore, resides in no one element. It is, essentially, the relation in which each factor of the constitution stands to the whole. That is to say, it resides only in the organised whole acting qua organised whole. If, for example, we speak of the “Sovereignty of the People” in a sense opposed to the Sovereignty of the State—
- 77. as if there were such a thing as “the people” over and above the organised means of expressing and adjusting the will of the community—we are saying what is, strictly speaking, meaningless. It is just the point of difference between Rousseau’s two views. We saw that Rousseau clearly explained the impossibility of expressing the general {283} will except by a determinate system of law. But what he seemed to suggest, and was taken to mean, by popular Sovereignty, was no doubt just the view which Hegel condemns. It is essentially the same question as how a constitution can be made. Strictly, a constitution cannot be made except by modification of an existing constitution. If, to put a case, you have a multitude new to each other in some extra-political colony, they must assume a constitution, so to speak, before they can make one. Law and constitution are utterances of the spirit of a nation. The form of State which Hegel analyses is a modern constitutional monarchy, with an executive (ministers sitting in the chambers, as he is careful to urge) and Chambers or Estates representing the classes developed in the civic community. Representation, he urges, is of bodies or interests rather than of masses of individuals, and the Corporations or Trade Societies have also an important place directly, by their touch with the departments of the executive government. [1] The general principle is, as indicated above, that the problems of connection between considerable particular interests and the universal interests of the community are, so to speak, prepared on the ground of the Corporation and Bourgeois Society for a solution in the interest of the common good by the Legislative and the Executive Government. [1] Much as through inspectors and commissions the opinion of Trade Unions, Friendly Societies, and Co-operators is elicited by our
- 78. Government Departments with a view to legislation, independently of the House of Commons. The logical division of power, in his language, {284} is that the Legislature has to establish universal principles, the executive has to apply these principles to particular cases, and the prince has to bring to a point the acts of the State by giving them, “like the dot on the i,” the final shape of individual volition. The distinction of States into Monarchy, Aristocracy, and Democracy, Hegel refuses to regard as applicable to the modern world. At best, it could only apply to the undeveloped communities of antiquity. The modern State is a concrete, and, according to its principle, all the elements of a people’s life are represented in it as an indivisible unity. A curious point is Hegel’s insistence on the function of the personal Head of the State. By a junction of the extremes, he connects it with the recognition of free individuality, which is usually regarded as the democratic principle of the modern world. There is no act, we may say in illustration, according to the modern idea of an act, if it is not done in the end by an individual, though in a developed political system the monarch’s action may only consist in signing his name. It is at least remarkable to compare this view with the tendency to one-man government in the administration of the United States of America. The State, then, is on one side the external force and automatic machinery implied in the maintenance and adjustment of the rights and purposes of the family and the Bourgeois Society as an actual life. On the other side, and most essentially, it is that connection of feeling and insight, working throughout the consciousnesses of
- 79. {285} individuals as parts in a connected structure, which unite in willing a certain type of life as a common good in which they find their own. It has the same content as that of Religion; but in an explicit and rationalised form as contrasted with the form of feeling. Only the separation of Church and State, and the division of the Churches against one another, have made it possible for the State to exhibit its own free and ethical character in true fulness, apart from both dogmatic authority and anarchic fanaticism. 9. Publicity of discussion in the assembly of the classes or estates is the great means of civic education. It is not in the least true that every one knows what is for the good of the State, and has only to go down to the House and utter it. It is in the work of expression [1] and discussion that the good takes form by adjustment of private views to facts and needs brought to bear by criticism. “The views a man plumes himself on when he is at home with his wife and friends are one thing; it is quite another thing what happens in a great assembly, where one shrewd idea devours the other.” [2] [1] It is a remarkable point in English politics to-day that legislation is practically in the hands of the Government departments. Bills are rejected or “knocked about in Committee”; but the mass of organised knowledge necessary to initiate legislation in a complex society can hardly be found outside the gathered experience of an office which has continuity in dealing with the same problems. This tendency more than justifies Hegel’s point of view. An act of the “General Will” has not only, as he said, to be moulded by running the gauntlet of public and critical discussion, but has even to be first drafted by the help of immense piles of experience, which the general mind does not possess, and could not deal with,
- 80. but which, nevertheless, enable its typical wish and intention to be embodied in effective form. [2] Rechtsphil., sect. 315. {286} The free judgment of individuals based on the publicity of political discussion is “public opinion.” In public opinion we have an actual existent contradiction. As public, it is sound and true, and contains the ethical spirit of the State. As expressed by individuals in their particular judgments, on which they plume themselves, it is full of falsehood and vanity. It is the bad which is peculiar, and which people pride themselves on; the rational is universal in its nature, though not necessarily common. Public opinion is a contradictory appearance, in which the true exists as false. It is no accident, but inevitable insight, that leads both of these characters to be proverbially expressed, as in “Vox populi, vox Dei,” contrasted with Ariosto’s “Che’l Volgare ignorante ogn’un’ riprenda E parli plu di qual che meno intenda”; [1] or Goethe’s “Zuschlagen kann die Masse Da ist sie respektabel; Urtheilen gelingt ihr miserabel.” [2] or the “mostly fools” of Carlyle. [1] “That the ignorant vulgar reproves everyone, and talks most of what it understands least.”
- 81. [2] “The masses are respectable hands at fighting, but miserable hands at judging.” Now, as public opinion thus combines truth and falsehood, the public cannot be in earnest with both, i.e. both cannot be its real will. But if we restrict ourselves to its express utterance, we cannot possibly tell what it is in earnest with—because it does not know. Therefore, the degree of passion {287} with which a given opinion is maintained throws no light on the question, on what points the public is really in earnest, in the sense of the “real will.” This can only be known from the substantive reality, which is the “true inwardness” of public opinion. This substantive reality, the true merits of any case, is not to be got by the study of mere public opinion as expressed, but when it is successfully divined and asserted, public opinion will always come round to it. If we ask how it is to be divined or known, we must go back to the analogy of a theory. The solution must be constructed so as to satisfy the real facts or needs, and the real facts or needs only become known in proportion as it is constructed, just as in scientific discovery. The man who can see and do what his age wills and demands is the great man of the age. Public opinion, then, demands to be at once esteemed and contemned; esteemed in its essential basis, contemned in its conscious expression. It is, however, the principle of the modern world that every one is allowed to contribute his opinion. When he has contributed it, and so far satisfied the impulse of self-assertion, he is likely to acquiesce in what is done, to which, he can feel, he has thrown in some element of suggestion or criticism. 10. In concluding this chapter, we will attempt to estimate the nearness of such an analysis of the State to the actual facts of life,
- 82. admitting certain appearances against it, but rejecting pessimistic views which rest on false abstractions. I will state the difficulties as they appeared to T.H. Green, a cautious and practical Englishman, {288} well experienced in local politics, and acquainted with different classes of men. [1] “To an Athenian slave, who might be used to gratify a master’s lust, it would have been a mockery to speak of the State as a realisation of freedom; and perhaps it would not be much less to speak of it as such to an untaught and under-fed denizen of a London yard with gin shops on the right hand and on the left.” “It is true that the necessity which the State lays on the individual is for the most part one to which he is so accustomed that he no longer kicks against it; but what is it, we may ask, but an external necessity, which he no more lays on himself than he does the weight of the atmosphere or the pressure of summer heat and winter frosts, that compels the ordinary citizen to pay rates and taxes, to serve in the army, to abstain from walking over the Squire’s fields, snaring his hares, or fishing his preserved streams, to pay his rent, to respect those artificial rights of property which only the possessors of them have any obvious interest in maintaining, or even (if he is one of the proletariate) to keep his hands off the superfluous wealth of his neighbour when he has none of his own to lose?” “A conception does not float in the air. It must be somebody’s conception. Whose conception, then, of general good is it that these institutions represent?”
- 83. “Is it not seriously misleading, when the requirements of the State have so largely arisen out of force directed by selfish motives, and when the motive of obedience to these requirements is determined by fear, to {289} speak of them as having a common source with the morality of which it is admitted that the essence is to be disinterested and spontaneous?” [1] Principles of Political Obligation, p. 8; cf. p. 127 ff. I have quoted these passages—the whole section should be carefully read—in order to state plainly a paradox which affects the theory of society from beginning to end. It continually shows itself in the pessimistic criticism of economic motive, political motive, and of every-day social motive. The whole question really depends on our understanding of the relation of abstract and concrete. It is plain, as Green says, that the idea of a common good has never been the sole influence operative in the formation or maintenance of States. And, in as far as it has operated at all, it has only done so in very imperfect forms. Green goes so far as to say that Hegel’s account of freedom as realised in the State does not seem to correspond to the facts of society as it is, or even as, under the unalterable conditions of human nature, it ever could be; though, no doubt, there is a work of moral liberation, which society, through its various agencies, is constantly carrying on for the individual. Now, the truth of these criticisms may be granted in the same sense in which we grant the imperfection of knowledge (as currently conceived) or of morality—imperfections not accidental, but inherent in each particular form of human experience. The conflict of interests, the failure to reconcile rights, and the weight and
- 84. opaqueness, so to speak, of law and custom to the individual mind, are contradictions of the same type and {290} due to causes of the same kind as those which arise in the world of ethics and of theory. And, though the new relations which spring up in society are perpetually resulting in new contradictions, there is no reason to compare the State unfavourably, in this respect, with Morality or with Science. The contradictions, in fact, are the material of organisation. [1] [1] Take, for instance, the chaos of the medical charities of London. It consists of endeavours to adjust help to needs, which endeavours are themselves unadjusted to each other. Thus, precisely as in the theoretical progress, the unadjustment of adjustments brings out ever new contradictions which demand readjustment. Without differing profoundly from Green in theory, therefore, we venture to assign a greatly diminished importance to his criticisms. This is due in part to the growth of a more intimate experience, owing in some measure to his initiative, which seems to show the essentials of life to be far more identical throughout the so-called classes of society than is admitted by such a passage as that cited above about the dweller in a London yard. [1] It is due, further, and in connection with such experience, to the psychological conceptions developed in previous chapters, according to which the place of actual fear of punishment in maintaining the social system is really very small, while {291} the place of a habituation, which is essentially ethical, is comparatively large. These suggestions, which lead us to lay decreasing stress on Green’s criticism of Hegel, point wholly in the general direction of his own convictions, and we may finally meet the general difficulty, which expresses itself in
- 85. Welcome to Our Bookstore - The Ultimate Destination for Book Lovers Are you passionate about books and eager to explore new worlds of knowledge? At our website, we offer a vast collection of books that cater to every interest and age group. From classic literature to specialized publications, self-help books, and children’s stories, we have it all! Each book is a gateway to new adventures, helping you expand your knowledge and nourish your soul Experience Convenient and Enjoyable Book Shopping Our website is more than just an online bookstore—it’s a bridge connecting readers to the timeless values of culture and wisdom. With a sleek and user-friendly interface and a smart search system, you can find your favorite books quickly and easily. Enjoy special promotions, fast home delivery, and a seamless shopping experience that saves you time and enhances your love for reading. Let us accompany you on the journey of exploring knowledge and personal growth! ebookgate.com