![2 1 Introduction
approach applies task-level metric and aims to serve one task at a time with
synchronized network access. While serialization is a smart design to avoid task-
level interference, our study shows that the flow level network contention within
a task can however largely affect the task completion time. This prolongs the tail
as well as the average task completion time and unavoidably reduces the system
applicability to serve delay-sensitive applications.
Containerization [1] has become a popular virtualization technology due to
many promising properties such as lightweight, scalable, highly portable, and good
isolation, and the emergence of software containerization tools, e.g., docker [2],
further allows users to create containers easily on top of any infrastructure.
Therefore, more and more Internet service providers are deploying their services
in the form of containers in modern data centers.
For large-scale online service providers, such as Google, Facebook, and Baidu,
an important data communication pattern is inter-DC multicast of bulk data –
replicating massive amounts of data (e.g., user logs, web search indexes, photo
sharing, blog posts) from one DC to multiple DCs in geo-distributed locations. Our
study on the workload of Baidu shows that inter-DC multicast already amounts to
91% of inter-DC traffic, which corroborates the traffic pattern of other large-scale
online service providers [3, 4]. As more DCs are deployed globally and bulk data
are exploding, inter-DC traffic then needs to be replicated in a frequent and efficient
manner.
While there have been tremendous efforts toward better inter-DC network
performance (e.g., [3, 5–9]), the focus has been improving the performance of the
wide area network (WAN) path between each pair of DCs. These WAN-centric
approaches, however, are incomplete, as they fail to leverage the rich application-
level overlay paths across geo-distributed DCs, as well as the capability of servers
to store-and-forward data. As illustrated in Fig. 1.1, the performance of inter-DC
multicast could be substantially improved by sending data in parallel via multiple
overlay servers acting as intermediate points to circumvent slow WAN paths and
performance bottlenecks in DC networks.
DC A DC B
DC C
DC B
DC C
DC A
(a) Direct replication over
pair-wise inter-DC WANs
(b) Replication leveraging
overlay paths
1
2
2 2
1
2
Overlay servers
Fig. 1.1 A simple network topology illustrating how overlay paths reduce inter-DC multicast
completion time. Assume that the WAN link between any two DCs is 1 GB/s, and that A wants to
send 3 GB data to B and C. Sending data from A to B and C separately takes 3 s (a), but using
overlay paths A → B → C and A → C → B simultaneously takes only 2 s (b). The circled
numbers show the order for each data piece is sent](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-15-320.jpg)
![1.2 Content Summary 3
It is important to notice that these overlay paths should be bottleneck-disjoint;
that is, they do not share common bottleneck links (e.g., A → B → C and A →
C → B in Fig. 1.1) and that such bottleneck-disjoint overlay paths are available in
abundance in geo-distributed DCs.
In recent years, many short video platforms are developing at an incredible
speed (such as Kuaishou [10], Douyin/TikTok [11], YouTube Go [12], Instagram
Stories [13], and so on), these platforms allow users to record and upload very short
videos (usually within 15 s). As a result of the massive short videos uploaded by
distributed users, caching problem is becoming more challenging compared with the
traditional centralized Content Delivery Network (CDN) whose traffic is dominated
by some popular items for a long period of time [14]. To handle scalability
and improve users’ Quality of Experience (QoE), the emerging edge computing
naturally matches the demand for distributed storage, and the abovementioned
platforms thus have resorted to employ edge caching servers to store and deliver
the massive short videos, so as to avoid that all requests have to be fetched from the
backend/origin server, which usually introduces extra user-perceived latency.
In recent years, with the rapid development of Internet of things technology,
a variety of smart devices that can be connected to the network have emerged.
However, these smart devices are different from the traditionals, especially some
of these kinds are highly mobile, such as the rapid driving vehicles in the Internet
of Vehicles, which led to the Internet of Vehicles becoming a kind of dynamic
networks. In fact, dynamic networks exist in all aspects of our lives, such as delay-
tolerant networks, opportunistic-mobility networks, social networks, friendship
networks, etc. [15, 16]. The topology of these networks is not static, but changes
over time, where the connections among nodes establish or destruct irregularly.
This results in rapidly changing network topology, which makes it difficult to hold
controllability of the entire network. To address this challenge, it is necessary to
propose a way to analyze and solve the controllability of dynamic networks.
1.2 Content Summary
In this chapter, we present the measurement and analysis of interactive workloads
on Baidu’s cloud platform. The real-world trace indicates that these interactive
workloads are suffering from significantly long latency than the general non-
interactive workloads. A typical case is shown in Fig. 1.2a where Nuomi is a group
buying application, Waimai is a take-out service, and Alipay is an online payment
platform. When a user clicks an item on Nuomi, the latency is quite short because
this query does not require many interactions among services. However, the story
will be different when the user orders a take-out and purchase the item. In this case,
the request goes through Nuomi, Waimai, and then Alipay. In other words, this
interactive workload consists of several highly dependent operations which have
to be processed on different servers separately, like Fig. 1.2b, and there are six
procedures for interactive workloads and only two procedures for non-interactive](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-16-320.jpg)
![4 1 Introduction
(a) (b)
Service 1
Service 2
Service 3
User Request
User 3
User 1
User 3
Non-interactive
Interactive
Data flow
User 2
nuomi.com
waimai.baidu.com
alipay.com
Fig. 1.2 The processes of interactive and non-interactive workloads
workloads. It is easy to see that such interactive workloads in chained applications
will introduce extra latency to users because these requests will be handled by
different services for multiple times.
Unfortunately, we find that most of the existing workload scheduling approaches
are aimed to reschedule [17] and leverage different priorities [18, 19] on individual
queues. In other words, these optimizations are made on intermediate servers
separately, so the overall latency of interactive workloads is still unpredictable.
To better optimize the overall latency of chained services, we apply a latency
estimation approach to predict overall latency and try to accelerate the interactive
workloads. Furthermore, we design a feedback scheme to ensure workload fairness
and avoid remarkable degradation of non-interactive workloads. Our real-world
deployments on Baidu indicate that the proposed delay-guarantee (D3G) framework
can successfully reduce the latency of interactive applications with minimal effect
on other workloads.
In this chapter, we for the first time investigate the potential to consider both flow
level and task level interference together for data center task scheduling. We provide
the design of TAFA (task-aware and flow-aware) to obtain better serialization
and minimize possible flow and task contentions. TAFA adopts dynamic priority
adjustment for the task scheduling. Different from FIFO-LM [20], this design
can successfully emulate shortest-task-first scheduling while requires no prior
knowledge about the task. Further, TAFA gives a more reasonable and efficient
approach to reduce task completion time by considering the relationship among
different flows in one task, rather than treating them all the same. With this
intelligent adjustment in flow level, TAFA provides the shorter flow waiting time and
leading to earlier finish time. As the total waiting time of a task consists of waiting
time and processing time, TAFA for the first time combines the two aspects together.](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-17-320.jpg)
![1.2 Content Summary 5
To achieve shorter processing time, TAFA solves the dominant resource matching
problem and classifies different requirements and VMs into different resource
dominant type. By allocating different resource dominant VMs to corresponding
resource-dominant requirements, we can achieve more rapid processing time.
Generally, each Internet service has several modules which are instantiated as a
set of containers, and the containers belonging to the same service often need to
communicate with each other to deliver the desired service [21–24], resulting in
heavy cross-server communications and downgrading service performance [19, 21].
If these containers are placed on the same server, the communication cost can
be greatly reduced. However, the containers belonging to the same service are
generally intensive to the same resource (e.g., containers of the big data analytics
services [25–27] are usually CPU-intensive, and containers of the data transfer
applications [4, 28–31] are usually network I/O-intensive). Assigning these contain-
ers on the same server may cause heavily imbalanced resource utilization of servers,
which could affect the system availability, response time, and throughput [32, 33].
First, it prevents any single server from getting overloaded or breaking down, which
improves service availability. Second, servers usually generate exponential response
time when the resource utilization is high [34]; load balancing guarantees acceptable
resource utilizations for servers, so that the servers can have fast response time.
Third, no server will be a bottleneck under balanced workload, which improves the
overall throughput of the system.
Figure 1.3 shows an example. Suppose there are two services (denoted by SA
and SB) to be deployed on two servers. Each service has two containers (CA1, CA2,
and CB1, CB2, respectively). The containers of SA are CPU-intensive, while the
containers of SB are network I/O-intensive. Figure 1.3a shows a solution which
assigns one of SA’s containers and one of SB’s containers on each server. This
approach achieves high resource utilization on both CPU and network I/O, but
incurs high communication cost between the two servers. Figure 1.3b shows another
solution where the containers of the same service are assigned on the same server.
The communication overhead is thus significantly reduced; however, the utilization
of CPU and network I/O is highly imbalanced on the two servers.
Server
CPU I/O
CA2 CA2 CA1
Server
CPU I/O
CB2
CB1
CB2
CB1
Network
CA1
(a)
Server
CPU I/O
CA2
CA1
CA2
CA1
Server
CPU I/O
CB2
CB1
CB2
CB1
Network
(b)
Fig. 1.3 The conflict between container communication and server resource utilization. (a)
High network communication overhead, balanced resource utilization. (b) Imbalanced resource
utilization, low network communication overhead](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-18-320.jpg)
![1.2 Content Summary 7
in other contexts (e.g., [35–38]), building one for inter-DC multicast traffic poses
two challenges. First, as each DC has tens of thousands of servers, the resulting
large number of possible overlay paths makes it unwieldy to update overlay routing
decisions at scale in real time. Prior work either relies on local reactive decisions by
individual servers [39–41], which leads to suboptimal decisions for lack of global
information, or restricts itself to strictly structured (e.g., layered) topologies [42],
which fails to leverage all possible overlay paths. Second, even a small increase
in the delay of latency-sensitive traffic can cause significant revenue loss [43], so
the bandwidth usage of inter-DC bulk-data multicasts must be tightly controlled to
avoid negative impact on other latency-sensitive traffic.
To address these challenges, BDS+ fully centralizes the scheduling and routing
of inter-DC multicast. Contrary to the intuition that servers must retain certain
local decision-making to achieve desirable scalability and responsiveness to net-
work dynamics, BDS+’s centralized design is built on two empirical observations
(Sect. 6.2): (1) while it is hard to make centralized decisions in real time, most
multicast data transfers last for at least tens of seconds and thus can tolerate slightly
delayed decisions in exchange for near-optimal routing and scheduling based on
a global view; (2) centrally coordinated sending rate allocation is amenable to
minimizing the interference between inter-DC multicast traffic and latency-sensitive
traffic.
The key to making BDS+ practical is how to update the overlay network in near
real-time (within a few seconds) in response to performance churns and dynamic
arrivals of requests. BDS+ achieves this by decoupling its centralized control into
two optimization problems, scheduling of data transfers and overlay routing of
individual data transfers. Such decoupling attains provable optimality and, at the
same time, allows BDS+ to update overlay network routing and scheduling in a
fraction of second; this is four orders of magnitude faster than solving routing and
scheduling jointly when considering the workload of a large online service provider
(e.g., sending 105 data blocks simultaneously along 104 disjoint overlay paths).
In practice, there is always a fixed upper bound of available bandwidth for
bulk-data multicast, because multicast overlay network shares the same inter-DC
WAN with online latency-sensitive traffic. Existing solutions always reserve a fixed
amount of bandwidth for the latency-sensitive traffic, according to its peak value.
This guarantees the strict bandwidth separation, but the side effect is the waste of
bandwidth, especially when the online traffic is in its valley. To further improve link
utilization, BDS+ implements dynamic bandwidth separation that can predict online
traffic and reschedule bulk-data transfer. In other words, BDS+ achieves dynamic
bandwidth separation between bulk-data multicast and online traffic to further speed
up data transfer.
We have implemented a prototype and integrated it in Baidu. We first deployed
BDS+ in 10 DCs and ran a pilot study on 500 TB of data transfer for 7 days
(about 71 TB per day). Our real-world experiments show that BDS+ achieves 3–
5× speedup over Baidu’s existing solution named Gingko, and it can eliminate
the incidents of excessive bandwidth consumption by bulk-data transfers. Using
micro-benchmarking, we show that BDS+ outperforms techniques widely used in](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-20-320.jpg)
![8 1 Introduction
CDNs, BDS+ can handle the workload of Baidu’s inter-DC multicast traffic with one
general-purpose server, and BDS+ can handle various failure scenarios.1 We then
use trace-driven simulations to evaluate BDS+ with dynamic bandwidth separation;
the results show that BDS+ further speeds up the bulk data transfer by 1.2 to 1.3
times in the network where online and offline services are mixed deployed.
There have been tremendous efforts toward better caching performance in
traditional CDN, and these caching algorithms can be classified into two categories:
the simple but effective reactive caching algorithms, such as First-In First-Out
(FIFO), Least Recently Used (LRU), Least Frequently Used (LFU), k-LRU, and
their variants, and the proactive caching algorithms, such as DeepCache [44]. These
prior solutions work well in traditional centralize-controlled CDN, but become
invalid in the emerging short video network due to the following two essential
differences.
First, the non-stationary user video access pattern. The basic assumption of
reactive caching policies is the stationary user access pattern, i.e., recently requested
or frequently requested content in the past should be kept in cache because these
policies assume that such contents have greater chance of being visited in the
future. While a study in [14] shows that in short video network, popular content
will become expired very quickly (within tens of minutes), indicating that the
popularity in the past could not represent that in the future, and this is the root cause
of the failure of these reactive policies (see Sect. 7.1.1). Second, The temporal
and spatial video popularity pattern, i.e., the change of video popularity, varies
in different edge caching servers during different time periods. Our study on the
workload of Kuaishou shows that it takes less than 1 h for a popular video to become
unpopular during peak hours while takes more than 3 h during late midnight.
Existing proactive caching policies that try to predict future content popularity
always focus on a fixed sight, making them fail in edge caching scenarios.
To address the above challenges, this chapter presents AutoSight, a distributed
caching mechanism that works in edge caching servers for short video network.
AutoSight allows edge servers to retain respective local caching sight to adapt to
local video access pattern and video life span. AutoSight’s distributed design is
built on two empirical observations: (1) although the historical video access data
is non-stationary on individual edge server (see Fig. 7.1), making it difficult to make
popularity prediction, there is sufficient correlations among videos within the same
edge server, because users tend to request related videos, contributing much cross
visits in edge servers, and thus improves distributed predictions, and (2) temporal
and spatial video popularity pattern brings challenge for future sight of caching
policy (see Fig. 7.2), but distributed design allows adaptive future sights, enabling
edge servers to make decisions according to different video expiration speeds.
1As the existing solutions are with fixed bandwidth separation, so in these series of experiments,
we use BDS+ without dynamic bandwidth separation as comparation, while BDS+ with dynamic
bandwidth separation is evaluated separately.](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-21-320.jpg)
![10 1 Introduction
This chapter is extended from our preliminary work [45] with significant
improvements including:
• We disclose a new problem (i.e., the container placement problem) that places a
set of newly instantiated containers into a data center, which is a necessary and
important phase in services’ life cycle.
• We propose the CA-WFD algorithm to solve the container placement problem
and conduct extensive experiments to evaluate the performance.
• We refine the algorithms proposed for the container reassignment problem and
significantly extend the experimental study for this problem.
Our contributions are summarized as follows:
• Characterizing Baidu’s workload of inter-DC bulk-data multicast to motivate the
need of application-level multicast overlay networks (Sect. 6.1).
• Presenting BDS+, an application-level multicast overlay network that achieves
near-optimal flow completion time by a centralized control architecture
(Sects. 6.2 and 6.3).
• Making dynamic bandwidth separation to further improve link utilization in
the network where online and offline services are mixed deployed (Sects. 6.2
and 6.4).
• Demonstrating the practical benefits of BDS+ by a real-world pilot deployment
and large-scale simulations in Baidu (Sects. 6.5 and 6.6).
Our contributions are summarized as follows:
• Characterizing Kuaishou’s workload of short video network to motivate the need
of distributed edge caching policy.
• Presenting AutoSight, a distributed edge caching mechanism working in edge
caching servers for short video network, which solves the problem of non-
stationary user access pattern and temporal/spatial video popularity pattern.
• Demonstrating the practical benefits of AutoSight by real traces.
First, this chapter raised some issues of controllability challenges faced by
dynamic networks; then took the vehicle network on a road as an example with the
moving speed, communication radius, average density of vehicles, and the control
time as different variable parameters; and designed an algorithm to calculate the
optimal solution of the minimum number of driver nodes required for a dynamic
network. Finally, after the experiment of simulation data, we got the influence of
the vehicle’s moving speed, communication radius, average density, and the control
time on the number of required driver nodes of a vehicle network. By deploying
a minimum of driver nodes to ensure the controllability of the entire network,
and maximizes resource conservation, improves efficiency, ultimately, guides the
deployment of mobile networks.](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-23-320.jpg)
![Chapter 2
A Survey of Resource Management
in Cloud and Edge Computing
Abstract This chapter is to summarize the processing of the business, starting from
the service access to the data center, to the data transmission control, to the server
back-end communication, and the data synchronization service support, tracking the
complete service flow of the data flow. And carry out comprehensive and in-depth
research work for each of these links.
2.1 Latency Optimization for Cloud-Based Service Chains
As more applications are deployed on clouds for better system scalability and lower
operation cost, service chains are developing quickly. Many studies have shown that
the latency is particularly problematic when interaction latency occurs together with
network delays [1].
To minimize the latency on cloud-based applications, many researchers focused
on minimizing datacenter latency, which does advance the state of the art [2]. We
can classify these literatures into two categories. First, some literatures focused on
network and processing latency. For example, Webb et al. [3] proposed a nearest
server assignment to reduce the client-server latency. Vik explored the spanning
tree problems in a distributed interactive application system for latency reduction
in [4]. Moreover, [5] and [6] introduced a game theory into this topic and modeled
the latency problem in DC as a bargaining game, and Seung guaranteed resource
allocation by proposing resource allocation [7]. The other kind of related work is
web service, which is an application model for decentralized computing and an
effective mechanism for data and service integration on the Web [8]. Web service is
becoming relatively mature recently. Some studies succeeded in dissecting latency
contributors [9], showing that back-office traffic accounts for a significant fraction
of web transactions in terms of requests and responses [10].
Although the above studies have already made excellent latency optimization,
they ignored the interactions among multiple services (e.g., the case in Fig. 1.2), and
the interactive workloads are suffering from longer latencies due to the processing
procedures on multiple intermediate servers. Many researchers investigated the
latency for interactive applications, and their researches showed that although these
© Springer Nature Singapore Pte Ltd. 2020
Y. Zhang, K. Xu, Network Management in Cloud and Edge Computing,
https://doi.org/10.1007/978-981-15-0138-8_2
15](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-27-320.jpg)
![16 2 A Survey of Resource Management in Cloud and Edge Computing
applications are quite delay-sensitive, service performance is greatly affected by
interactions. To address this problem, some studies suggested that the interactions
of different services should be further dissected to better understand the performance
implications [10], and some researchers have already began to pay attention to
interaction latency [11].
Our study explores the potential to reduce the response time for service chains
and guarantee the non-interactive workloads simultaneously. In particular, we
accelerate the interactive workloads by building a new dedicated queue and try
to adjust resource allocation among different queues. By leveraging a feedback
scheme, we can bound the influence on non-interactive workloads. We’ll describe
the algorithm in Sect. 3.2 in detail after introducing our motivation in Sect. 3.1.
2.2 Toward Shorter Task Completion Time
Although the ever-increasing used data center networks have configured high
bandwidth and calculative ability, the task completion time still can be reduced to
a large extent [12]. In this chapter, we describe the nature of today’s datacenter
transport protocols, either flow-aware or task-aware and how the awareness of
different levels does isolate with each other. As a result, although flow completion
time or task completion time seems to reduce obviously, flow-aware protocols are
blind to task level and vice versa. In particular, good flow-level awareness can help
make task completion time shorter, while good task-level awareness can help flows
cooperate harmonically.
Figure 2.1 shows the development history of scheduling protocols, from DCTCP
(2010) to FIFO-LM (2014), which can be categorized into two broad classes, flow-
aware and task-aware, both of which do advance the state-of-the-art technique. We’ll
give a brief description and explanation to this progress.
As the founder of many flow-aware TCP-like protocols, DCTCP [13] leverages
Explicit Congestion Notification (ECN) in the network to provide feedback to end
hosts. Experiments show that DCTCP delivers better throughput than TCP while
2010 2011 2012 2013 2014 2015
Time
Flow
Aware
Task
Aware
Task&Flow
Aware
DCTCP D3 PDQ
D2TCP
pFabric
pase
FIFO-LM
TAFA
Fig. 2.1 Brief development history of transport protocols](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-28-320.jpg)
![2.2 Toward Shorter Task Completion Time 17
using 90% less buffer because it elegantly reduces the queue length. However, it is a
deadline-agnostic protocol that equally throttles all flows, irrespective of whether
their deadlines are near or far, so it may be less effective for the online data-
intensive (OLDI) applications [14]. Motivated by these observations, D3 [15] use
explicit rate control for the datacenter environment according to flow deadlines.
D3 can determine the rate needed to satisfy the flow deadline when having the
knowledge of flow’s sizes and deadlines. Although it outperforms TCP in terms of
short flow latency and burst tolerance, D3 has the practical drawbacks of requiring
changes to the switch hardware, which makes it not able to coexist with legacy
TCP [14]. Deadline-Aware Datacenter TCP (D2TCP) [14] is a deployable transport
protocol compared to D3. Via a gamma correction function, D2TCP uses ECN
feedback and deadlines to modulate the congestion window. Besides, D2TCP can
coexist with TCP without hurting bandwidth or deadline. Preemptive Distributed
Quick (PDQ) flow scheduling [16] is designed to complete flows quickly and meet
flow deadlines, and it builds on traditional real-time scheduling techniques: earliest
deadline first and shortest job first, which help PDQ outperform TCP, RCP [17],
and D3 significantly. pFabric [18] decouples flow scheduling from rate control.
Unlike the protocols above, in pFabric, each flow carries a single priority number
set independently, according to which switches execute a scheduling/dropping
mechanism. Although pFabric achieves near-optimal flow completion time, it does
not support work conservation in a multi-hop setting because end-hosts always send
at maximum rate. To make these flows back off and let a lower priority flow at a
subsequent hop, we need an explicit feedback from switches, i.e., a higher layer
control.
From a network perspective, tasks in DCNs typically comprise multiple flows,
which traverse different servers at different times. Treating flows of one task in
isolation will make flow-level optimizing while hurting task completion time. To
solve this boundedness in flow-aware schemes, task-aware protocols have been
proposed to explicitly take the higher layer information into consideration.
A task-aware scheduling was proposed by Fahad R. Dogar in [12]. Using First-
In-First-Out scheme to reduce both of the average and the tail task completion
time, Dogar implemented First-In-First-Out with Limited Multiplexing (FIFO-LM)
to change the level of multiplexing when heavy tasks are encountered, which can
help heavy tasks not being blocked, or even starved. But as we all know, FIFO
is not the most effective method to reduce average completion time no matter in
flow level or task level, and the simple distinguish just between elephant tasks and
mouse tasks is in coarse granularity as [19] said that the DCNs should be more load,
more differentiation. Further, [20] and [21] give methods that can ensure user-level
performance guarantee.
Without cross-layer cooperation, these protocols have great blindness to each
other, making scheduling inefficient. In order to give our method the advantages
from both flow level and task level, we praise TAFA with the idea of co-assist that
making flow-level schedule help task complete early and task-level schedule help
flows mutually related. Our work performs well even in multiple resource-sharing
environment.](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-29-320.jpg)
![18 2 A Survey of Resource Management in Cloud and Edge Computing
2.3 Container Placement and Reassignment
for Large-Scale Network
In this section, we survey some problems that are related to our problem, including
multi-resource generalized assignment problem, Google Machine Reassignment
Problem, traffic-aware virtual machine placement, network function placement, and
container deployment and migration.
Multi-resource Generalized Assignment Problem (MRGAP) MRGAP [22, 23]
is an extension of the Generalized Assignment Problem (GAP) [24, 25], where
multiple resources are associated with the items and bins. Solutions to MRGAP
usually contain two phases. The first phase aims to obtain an initial feasible
solution, and the second phase attempts to further improve the solution. Gavish
et al. [23] proposed two heuristics to generate the initial solution and a branch and
bound algorithm to improve the solution. Privault et al. [26] computed the initial
solution by the bounded variable simplex method and optimized the solution by
a simulated annealing algorithm. Mitrović-Minić and Punnen [27] and Yagiura et
al. [28] generated a random initial solution in the first phase and adopted local
searching techniques in the second phase. Mazzola and Wilcox [29] combined Pivot
and Complement (P&C) and the heuristic proposed in [23] to obtain high-quality
solutions. In [30], Shtub et al. proposed a gradient descent-based solution to the
dynamic MRGAP (DMRGAP), where the resource requirements of items change
over time and an item can be assigned to several bins. Although we show that
MRGAP is equivalent to the simplified CPP in Sect. 5.3.1, we emphasize that CPP
and CRP are more complex than MRGAP because of the containerization-specific
constraints (i.e., conflict, spread, co-locate, and transient constraints), which makes
above solutions inapplicable in our scenarios.
Google Machine Reassignment Problem (GMRP) GMRP was formulated by
the Google research team as a subject of ROADEF/EURO Challenge, which aims
to maximize the resource usage by reassigning processes among the machines
in data centers. Gavranović et al. proposed the winner solution [31] noisy local
search (NLS), which combines local searching techniques and noising strategy
in reallocation. Different from NLS, we depart the reassignment into two steps,
namely, Sweep and Search. With the help of Sweep, we mitigate the hot hosts
and obtain better initial conditions for the following local searching procedure. The
evaluation result in Sect. 5.6 shows that Sweep&Search yields significantly better
results than directly applying local searching techniques.
Traffic-Aware Virtual Machine Placement Like containerization, virtual
machine (VM) is also a popular virtualization technique, where isolated operation
systems run above a hypervisor layer on bare metals. Since each VM runs a full
operating system [32], VMs usually have bigger sizes and consume more power than
containers. Hence, traditional VM placement mainly concerns about optimization of
energy consumption, resource utilizations, and VM migration overhead [33]. Since
the pioneer work of [34], many efforts have been made to mitigate inter-server](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-30-320.jpg)
![2.3 Container Placement and Reassignment for Large-Scale Network 19
communications by traffic-aware VM placement [35–44]. Meng et al. [34] defined
the traffic-aware VM placement problem and proposed a two-tier approximate
algorithm to minimize inter-VM communications. Choreo [36] adopts a greedy
heuristic to place VMs to minimize application completion time. Li et al. [37]
proposed a series of traffic-aware VM placement algorithms to optimize traffic
cost as well as single-dimensional resource utilization cost. Rui et al. [42] adopt a
system optimization method to re-optimize VM distributions for joint optimization
of resource load balancing and VM migration cost. Different from this work,
we optimize both communication overhead and multi-resource load balancing.
Besides, since containers can be deployed in VMs instead of physical machines, the
solutions proposed in our book are orthogonal to these VM placement strategies.
Therefore, VM resource utilization and inter-VM communications could be
optimized by container placement/reassignment, and that of physical machines
could be optimized by VM placement.
Network Function Placement Network functions virtualization (NFV) has
recently gained wide attention from both industry and academia, making the study
of their placement a popular research topic [45–60]. Wang et al. [45] studied the
flow-level multi-resource load balancing problem in NFV and proposed a distributed
solution based on the proximal Jacobian ADMM (alternating direction method of
multipliers). Marotta et al. [49] proposed a mathematical model based on the robust
optimization theory to minimize the power consumption of the NFV infrastructure.
In [53], an affinity-based heuristic is proposed to minimize inter-cloud traffic
and response time. Zhang et al. [54] proposed a Best-Fit Decreasing-based
heuristic algorithm to place network functions to achieve high utilization of single
dimensional sources. Taleb et al. studied the network function placement problem
from many aspects, including minimizing path between users and their respective
data anchor gateways [55], measuring existing NFV placement algorithms [56],
placing Packet Data Network (PDN) Gateway network functionality and Evolved
Packet Core (EPC) in the cloud [57, 58, 60], and modeling cross-domain network
slices for 5G [59]. Since network functions work in chains and containers are
deployed by groups, the communication patterns are totally different between the
two systems. Hence, the communication optimization solutions in NFV are non-
applicable to container placement. Besides, none of these works aim at the joint
optimization of communication overhead and multi-resource load balancing in data
centers.
Container Deployment and Migration A lot of work has been studied to deploy
containers among virtual machines or physical machines for various optimization
purposes. Zhang et al. [61] proposed a novel container placement strategy for
improving physical resource utilization. The works [62–64] studied the container
placement problem for minimizing energy consumption in the cloud. Mao et al. [65]
presented a resource-aware placement scheme to improve the system performance in
a heterogeneous cluster. Nardelli et al. [66] studied the container placement problem
for optimizing deployment cost. However, none of the above work considers the
communication cost among containers.](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-31-320.jpg)
![20 2 A Survey of Resource Management in Cloud and Edge Computing
The container migration issues also have been extensively studied in the litera-
ture. The first part of the related works concentrates on developing container live
migration techniques. The works [67, 68] proposed solutions for live migrating
Linux containers, while [69] proposed the techniques for live migrating Docker
containers. The prior works [70–72] further optimized the existing container
migration techniques for reducing migration overhead. The second part of the
related works focused on the container migration strategies. Li et al. [73] aimed
to achieve load balancing of cloud resources through container migration. Guo et
al. [74] proposed a container scheduling strategy based on neighborhood division
in micro service, with the purpose of reducing the system load imbalance and
improve the overall system performance. Kaewkasi and Chuenmuneewong [75]
applied the ant colony optimization (ACO) in the context of container scheduling,
which aimed to balance the resource usages and achieve better performance. Xu
et al. [76] proposed a resource scheduling approach for the container virtualized
cloud environments to reduce response time of customers jobs and improve resource
utilization. Again, none of the above work considers communication cost among
containers.
2.4 Near-Optimal Network System for Data Replication
Here we discuss some representative work that is related to BDS+ in three
categories.
Overlay Network Control Overlay networks realize great potential for various
applications, especially for data transfer applications. The representative networks
include peer-to-peer (P2P) networks and content delivery networks (CDNs). The
P2P architecture has already been verified by many applications, such as live
streaming systems (CoolStreaming [77], Joost [78], PPStream [79], UUSee [80]),
video-on-demand (VoD) applications (OceanStore [81]), distributed hash tables
[82], and more recently Bitcoin [83]. But, self-organizing systems based on P2P
principles suffer from long convergence times. CDN distributes services spatially
relative to end users to provide high availability and performance (e.g., to reduce
page load time), serving many applications such as multimedia [84] and live
streaming [85].
We briefly introduce the two baselines in the evaluation section: (1) Bullet
[86], which enables geo-distributed nodes to self-organize into an overlay mesh.
Specifically, each node uses RanSub [87] to distribute summary ticket information
to other nodes and receive disjoint data from its sending peers. The main difference
between BDS+ and Bullet lies in the control scheme, i.e., BDS+ is a centralized
method that has a global view of data delivery states, while Bullet is a decentralized
scheme and each node makes its decision locally. (2) Akamai designs a three-
layer overlay network for delivering live streams [88], where a source forwards its
streams to reflectors and reflectors send outgoing streams to stage sinks. There are
two main differences between Akamai and BDS+. First, Akamai adopts a three-](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-32-320.jpg)
![2.4 Near-Optimal Network System for Data Replication 21
layer topology where edge servers receive data from their parent reflectors, while
BDS+ successfully explores a larger search space through a finer-grained allocation
without the limitation of three coarse-grained layers. Second, the receiving sequence
of data must be sequential in Akamai because it is designed for a live streaming
application. But there is no such requirements in BDS+, and the side effect is that
BDS+ has to decide the optimal transmission order as additional work.
Data Transfer and Rate Control Rate control of transport protocols at the DC
level plays an important role in data transmission. DCTCP [89], PDQ [90], CONGA
[91], DCQCN [92], and TIMELY [93] are all classical protocols showing clear
improvements in transmission efficiency. Some congestion control protocols like the
credit-based ExpressPass [94] and load balancing protocols like Hermes [95] could
further reduce flow completion time by improving rate control. On this basis, the
recently proposed NUMfabric [96] and Domino [97] further explore the potential
of centralized TCP on speeding up data transfer and improving DC throughput.
To some extend, co-flow scheduling [98, 99] has some similarities to the multicast
overlay scheduling, in terms of data parallelism. But that work focuses on flow-level
problems, while BDS+ is designed at the application level.
Centralized Traffic Engineering Traffic engineering (TE) has long been a hot
research topic, and many existing studies [100–106] have illustrated the challenges
of scalability, heterogeneity etc., especially on inter-DC level. The representative
TE systems include Google’s B4 [107] and Microsoft’s SWAN [108]. B4 adopts
SDN [109] and OpenFlow [110, 111] to manage individual switches and deploy
customized strategies on the paths. SWAN is another online traffic engineering
platform, which achieves high network utilization with its software-driven WAN.
Network Change Detection Detecting network changes is quite important not
only in traffic prediction problems, but also in many other applications, such as
abnormality detection, network monitoring, and security. There are two basic but
mature methods that are widely used, the exponentially weighted moving average
(EWMA) control scheme [112, 113] and the change point detection algorithm [114].
EWMA usually gives higher weights to recent observations while gives decreased
weights in geometric progression to the previous observations, when predicting
the next value. Although EWMA describes a graphical procedure for generating
geometric moving averages smoothly, it faces an essential sensitivity problem; in
other words, it cannot identify abrupt changes. In contrast, change point detection
algorithms could exactly solve this problem, in both online [115–117] and offline
[118–121] manner. BDS+ combines these two methods by designing a sliding
observation window, which makes BDS+’s prediction algorithm both stable and
sensitive.
Overall, an application-level multicast overlay network with dynamic bandwidth
separation is essential for data transfer in inter-DC WANs. Applications like user
logs, search engine indexes, and databases would greatly benefit from bulk-data
multicast. Furthermore, such benefits are orthogonal to prior WAN optimizations,
further improving inter-DC application performance.](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-33-320.jpg)
![22 2 A Survey of Resource Management in Cloud and Edge Computing
2.5 Distributed Edge Caching in Short Video Network
Here we discuss some representative caching policies in traditional CDN and some
related edge caching systems.
Existing caching policies The most representative caching policies such as FIFO,
LRU, LFU, and their variations are simple but effective in traditional CDN, where
the frequency of content visits can be modeled as Poisson distribution. Under these
policies, the future popularity of a content is represented by the historical popularity.
But in short video network, user access pattern is non-stationary and is no longer in
Poisson distribution, so these policies become inefficiency in short video network.
The same problem exists in the TTL (Time-to-live)-based caching policies. For
example, in the caching (feedforward) networks, where the access pattern is Markov
arrival process, [122] gives joint consideration about both TTL and request models
and drives evictions by stopping times. Ferragut et al. [123] set an optimal timer to
maximize cache hit rate, but it only works in the case of Pareto-distributed access
pattern and Zipf distribution of file popularity and thus certainly becomes invalid in
short video network. Basu et al. [124] propose two caches named f-TTL with two
timer, so as to filter out non-stationary traffic, but it still relies on locally observed
access patterns to change TTL values, regardless of the future popularity.
One of the promising attempts in recent years is learning-based proactive
prediction-based policy. Narayanan et al. [125] train a characteristics predictor to
predict object future popularity and interoperate with traditional LRU and LFU,
boosting the number of cache hits. However, it looks a fixed length into the future
to predict object popularity (1–3, 12–14, and 24–26 h), ignoring the temporal and
spatial video popularity pattern, thus cannot handle the varies life spans in short
video network. Pensieve [126] trains a neural network model as an adaptive bitrate
(ABR) algorithm; it complements our AutoSight framework, in the sense that the
proposed dynamic control rules can give us a reference when facing the various
life span problem, but it ignores temporal pattern information and only works in
live streaming. Besides, [127] introduces reinforcement learning for making cache
decisions, but it works in the case that user requests comply with the Markov
process, while the proposes AutoSight in this book can work under arbitrary non-
stationary user access patterns.
Edge caching systems Edge computing was proposed to enable offloading of
latency-sensitive tasks to edge servers instead of cloud and has achieved rapid
development in many areas such as 5G, wireless, mobile networks [128], and video
streaming [129]. Cachier [130] uses a caching model on edge servers to balance load
between edge and cloud for image-recognition applications, but it doesn’t predict
image loads in the future. Gabry et al. [131] study the content placement problem
in edge caching to maximize energy efficiency; the analysis in this work gives us
references to design our AutoSight network topology, but what we considered under
such topology is the caching of short video network rather than energy-saving.](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-34-320.jpg)
![2.6 The Controllability of Dynamic Temporal Network 23
Ma et al. [129] propose a geo-collaborative caching strategy for mobile video
network, suggesting that joint caching over multiple edges can improve QoE, which
provides strong proof for our AutoSight design. While this chapter tries to reveal the
characteristics of different mobile videos, we focus the short video network on edge
servers with unique user access pattern and video popularity pattern.
2.6 The Controllability of Dynamic Temporal Network
Dynamic network The network is composed of nodes and the relationship
between nodes. The rapid development of the Internet makes human beings enter
the network era, such as all kinds of social networks. The industrial Internet such
as large power grid and the Internet of things, connects the whole world into a
huge network. At present, network research has infiltrated into mathematics, life
science, information science, and many other fields, and its fundamental goal is
to find effective means to control network behavior and make it serve human
beings. The paper [132] firstly applies the classical control theory to the analysis
of network control. In a directed network, it uses a directed edge from node N1
to node N2 to denote that N2 can be controlled by N1 under some conditions.
In this book, the network is abstracted into a linear time-invariant system, and
the Kalman controllability criterion in the control theory is introduced into the
research, and the sufficient and necessary condition for the network to achieve
complete controllability is that the controllability matrix reaches full rank, so the
controllability problem of the network is transformed into the problem of calculating
the rank of the controllability matrix.
What is mentioned in [133] is theoretically proved that the minimum driver
nodes needed to control the entire network depend on the maximum matching
in the network, in which all unmatching nodes are driver nodes of the network.
Thus, the problem of network structure controllability is transformed into a classical
graph theory matching problem, which reduces the time complexity of network
controllability problem. However, this method can only be applied to directed
networks and networks with unknown edge weights. According to Popov-Belevitch-
Hautus criterion, the paper [134] has proved that the minimum number of driver
nodes required by the network is equal to the maximum geometric multiplicity of
all eigenvalues of the network matrix; as for undirected networks, the minimum
driver nodes are the maximum algebraic multiplicity of all eigenvalues. It reduces
the computational complexity of related problems greatly.
References [135–138] presented a theoretical approach to describing the control-
lability of networks or proposed a way to change the state of some nodes to stabilize
the network. All of the papers and methods mentioned above have not solved the
problem of controllability generated in dynamic networks well until now.
Internet of Vehicles Internet of Vehicles refers to the use of vehicle electronic
sensing devices to enable two-way data exchange and sharing between vehicles,](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-35-320.jpg)
![24 2 A Survey of Resource Management in Cloud and Edge Computing
Fig. 2.2 Illustration of vehicle-to-vehicle communication
people, and transportation facilities through wireless communication technology,
car navigation systems, intelligent terminal facilities, and information processing
systems. It is comprehensive intelligent decision-making information system that
realizes real-time monitoring, scientific dispatching, and effective management
of vehicles, people, objects, roads, etc., thereby improving road transportation
conditions and traffic management efficiency [139].
Vehicle-to-vehicle communication is shown in Fig. 2.2, the communication
between the vehicle and the vehicle through the vehicle-mounted terminal, and it is
mainly used for two-way data transmission between vehicles. The communication
technologies used include microwave, infrared technology, dedicated short-range
communication, etc., featuring high safety and real-time requirements. The vehicle
terminals can collect information such as the speed, position, direction, and vehicle
running alarm of the surrounding vehicles in real time. Those vehicles form an
interactive communication platform through wireless communication technology,
which can exchange pictures, text messages, videos, and audio information in real
time [140, 141] (Fig. 2.3).
The communication between the vehicle and the control center means that the
vehicle-mounted mobile terminal establishes interconnection with the remote traffic
control center through the public access network, to complete data transmission and
information exchange and to accomplish the interaction and storage of data between
the vehicle and the traffic control center. It is mainly used in vehicle navigation,
vehicle remote monitoring, emergency rescue, information entertainment services,
and so on. Moreover, it has the characteristics of long distance and high-speed
movement [139].
In a vehicular network, vehicles are virtualized as mobile network nodes, and
road side units (RSUs) are virtualized as stationary network nodes. The environmen-
tal information of the road and the vehicle is collected through sensors in the vehicle
and the RSU. The structure of the vehicle network presents a dynamic topology. The
high-speed moving vehicle nodes make the topology of the vehicle network change](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-36-320.jpg)
![saimme kuulla, mutta uteliaisuutemme oli kuitenkin hyvässä
vireessä, kun 10 p. heinäk. laittausimme taipaleelle sinne.
Vienanmerellä on joitakuita vuosia ollut säännöllinen
höyrylaivaliikenne, jonka eräs "Arkangelin-Muurmannin"
höyrylaivayhtiö Vienassa on pannut toimeen. Sitä ylläpiti tänä kesänä
kaksi yhtiön höyryä "Onega" ja "Kemi", joista edellinen kulki pitkin
Karjalan rannikkoa Kannanlahdesta aina Onegan kaupunkiin saakka,
jälkimmäinen Kemistä Vienaan; yhtymäpaikka oli Kemin reti. Sen
puolesta tämä kulku vielä kuitenkin oli vajanainen ja vähemmin
tyydyttävä, että laivat ainoastaan joka kymmenes päivä kohtasivat
toisensa, ja kun emme sattuneet tulemaan semmoiseen aikaan,
täytyi meidän soutuveneellä yrittää luostarisaareen.[11]
Ennenkuin päästiin lähtemään, alkoi kello jo lähetä 11 ap. Me
olimme kyllä aikoja olleet valmiit ja aloimme jo pitkäksyä odotusta,
mutta talonväen puolelta kesti yhä ahkeraa hommaa. Syyksi
viipymiseen sanottiin, että soutajat eivät olleet valmiit, mutta
todellinen syy lienee ollut, että talon kaksi lasta, poika ja tyttö,
myöskin laitettiin matkaan, vaikka siitä vasta rannassa kuulimme, ja
että isäntäväkeämme vaivasi saamattomuus, josta paluumatkalla
saimme loistavia todistuksia. Viimein astuttiin alas rantaan
yläpuolelle saarisiltaa, missä vene oli. Meitä karttui tässä sievä
joukko matkalle-lähtijöitä: kuusi vaimonpuolista soutajaa,
perämiehenä Huotari, Miikkula, jonka minä olin halunnut saada
oppaaksi mukaan, hänen turvissaan hänen kaksi näppärää, vaikka
vähän vallatonta veljenlastaan, me molemmat ja sitten eräs köyhä
kerjäläispoika, jolle perämies omasta mahtipontisuudestaan oli
antanut luvan tulla mukaan. Nämä 13 henkeä olivat näkyvissä, mutta
yhden ainakin vielä tiesimme olevan veneessä, sillä isäntämme oli
aamulla pyytänyt eräälle karjalaiselle tuttavalleen lupaa päästä](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-46-320.jpg)
![tuomiokirkko (Preobrashenskij sobor), jonka perustukset "iguumen"
eli apotti Filip laski v. 1558 ja jossa luostarin ensimmäisten
perustajain, Sauvatin ja Sosiman, komeat ruumisarkut säilytetään.
Kirkon portaille johtaa leveä, 30-40 syltä pitkä käytävä kartanon
poikki, joka on pensaita ja kukkasarkoja aivan täynnä, niin että se
näyttää mitä kauneimmalta kasvitarhalta. Kirkastuksen kirkko on
eteläosa laajasta rakennussarjasta, joka ulottuu kartanon
pohjoispuoleen asti. Paitsi erinäisiä muita huoneita, niinkuin esim.
kirjastoa, on tässä sarjassa kolme kirkkoa: lähinnä Kirkastuksen
kirkkoa, vähän edempänä, niin että se yhtyy itämuuriin, eräs kirkko,
jonka katto on sininen ja kultatähdillä koristettu (kaikki muut katot
ovat vihreitä) ja jonka Dixon (kirjassaan "Free Russia") sanoo
rakennetun erään englantilaisen laivaston karkoittamisen muistoksi,
mutta jota toinen niistä kirjoista, jotka luostarissa ostin, nimittää
piispa Filipin kirkoksi; senjälkeen piispa Nikolain kirkko ja viimein
pohjoisimpana Neitsyt Maarian taivaaseenastumisen kirkko
(Uspenskij sobor), jota käytetään refektoriona eli ruokasalina ja jota
talvella saatetaan lämmittää. Senkin rakennutti jo mainittu apotti
Filip vv. 1552-57 ja se lienee vanhin luostarin nykyisistä
rakennuksista. Nikolain kirkon vieressä on kellotapuli. Pohjoispuolella
tätä rakennussarjaa on avoin piha, jonka luoteispäässä on
gostinnitsan kohdalle tuleva portti ja jonka pohjoissyrjää reunustaviin
korkeihin rakennuksiin tehdyn toisen portin eli holvin kautta tullaan
korkeiden rakennusten välissä olevalle pienelle pihalle: se on vankien
piha, ja nuo korkeat rakennukset ovat vankihuoneita. Lukematon
joukko kyyhkysiä asuu täällä vankien parissa.[12]
Paitsi jo mainittuja neljää kirkkoa oli vielä yksi, erillänsä toisista ja
etelämuurissa kiinni. Se on varmaan se Maarian ilmestyksen kirkko,
joka rakennettiin kohta sen jälkeen kuin muurit luostarin ympäri oli
saatu valmiiksi (v. 1594).](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-60-320.jpg)
![luostarissa olevan kaikkiaan 500:n ja 1,000:n hengen välillä, mutta
pyhiinvaeltajia oli nyt arviolta kahdenvertainen määrä, s.o. toista
tuhatta henkeä. Joku osa niistä oli kotoisin rannikkoseuduista,
Karjalasta ja pohjois-Venäjältä, ja olivat omilla veneillään tänne
purjehtineet, mutta suurin osa oli tullut höyrylaivoilla Vienasta ja
olivat kotoisin sisä-Venäjältä. Venäläisten samoin kuin
muhamettilaisten uskontoon kuuluu nimittäin vaeltaminen pyhiin
paikkoihin, joita Venäjällä on koko joukko ja joista Solovetsin luostari
on mainioimpia. Kun niissä on jonkun "ihmeitätekevän" pyhänkuvan
edessä tehty ristinmerkkejä ja lyöty otsaa lattiaan, sitten suudeltu
pyhimysten ruumisarkkuja ja kuunneltu messua sekä ostettu
vahakynttilöitä palamaan jonkun kuvan eteen, on muka tehty
Jumalalle erittäin otollinen työ, josta autuuden porttia ainakin raolle
aukaistaan. Että tämmöiset pyhiinvaeltajat eli "bohomoltsit"
(rukoilijat) melkein yksinomaan ovat alhaista, sivistymätöntä kansaa,
on itsestään ymmärrettävää; sentähden mekin koko tässä
ihmislaumassa tuskin näimme kymmentä säätyhenkilöä. Että moni
heistä myöskin on laiskuri, joka kernaammin maleksii maita
mantereita kuin tekee työtä, voinee varmasti väittää, vaikka pääosa
lieneekin vilpittömästä uskonhartaudesta liikkeelle lähtenyt.[13]
Puolipäiväsaarnan eli messun toimituksessa oli kymmenkunta
ylhäistä pappia osallisena, kaikilla sentapaiset kultalankakankaasta
tehdyt kaavut yllään kuin meidän piispoilla kirkollisissa
juhlatilaisuuksissa. He seisoivat piirissä keskellä lattiaa; alimpana
joukossa, selin ovea ja päin alttaria kohti, oli pitkäpartainen,
kunnioitettavan näköinen vanhus, jota sanottiin väliaikaiseksi
arkkimandriitiksi. Varsinaista arkkimandriittia ei näet luostarissa ollut,
sittenkuin entinen oli joko kuollut tai nimitetty muuhun virkaan.
Pitkin ovenpuolista seinää istui erinäisillä istuimilla puolisensataa
munkkia. Kaikki meno näytti erittäin juhlalliselta, ja kun laulukunta](https://image.slidesharecdn.com/98544-250325023708-8ad8b5fc/85/Network-Management-in-Cloud-and-Edge-Computing-Yuchao-Zhang-64-320.jpg)
Network Management in Cloud and Edge Computing Yuchao Zhang
- 1. Read Anytime Anywhere Easy Ebook Downloads at ebookmeta.com Network Management in Cloud and Edge Computing Yuchao Zhang https://ebookmeta.com/product/network-management-in-cloud- and-edge-computing-yuchao-zhang/ OR CLICK HERE DOWLOAD EBOOK Visit and Get More Ebook Downloads Instantly at https://ebookmeta.com
- 2. Yuchao Zhang Ke Xu Network Management in Cloud and Edge Computing
- 3. Network Management in Cloud and Edge Computing
- 4. Yuchao Zhang • Ke Xu Network Management in Cloud and Edge Computing
- 5. Yuchao Zhang Beijing University of Posts and Telecomm Beijing, China Ke Xu Tsinghua University Beijing, China ISBN 978-981-15-0137-1 ISBN 978-981-15-0138-8 (eBook) https://doi.org/10.1007/978-981-15-0138-8 © Springer Nature Singapore Pte Ltd. 2020 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore
- 6. Preface Both cloud computing and edge computing have their specific advantages. This book addresses the challenges in both cloud and edge computing. Traditional cloud services are providing more and more convenient services for network users, while the resource heterogeneity and different server configurations bring serious challenges to the network performance. These challenges lie in the overall procedures of application processing. First, when an end user sends a request to one network application, the server should make access control and decide whether to accept it or not. Then the server should further schedule all the accepted requests and make transmission control. When this request is being served, it will call for server communications and coprocessing to satisfy the function requirements. For the request that needs cross-region services, different IDCs (Internet data centers) have to make information synchronization and data transmission. At last, the final calculation results could be sent back to this request. To provide high performance and shorter latency to network users, this book clarifies the overall procedures of network application requests. For each procedure, this book proposes a customized solution and optimization scheme. The emerging edge computing provides obvious advantages in the edge by enabling the computing and caching near end users. But due to the limitation of edges, the design principles are totally different from that in the cloud. This book focuses on both the calculation and storage problems and provides the general mechanisms to address these challenges. Overall, this book first analyzes the working procedure of cloud computing and edge computing and then provides detailed solutions for both trends. It can contribute to both traditional data center network and the emerging edge network. v
- 7. vi Preface Book Organization This book contains eight chapters organized in two logical parts. Despite the continuity and contrast relationship between each chapter, we tried to keep each chapter self-contained to maximum reading flexibility. Chapter 1 Data center has many features such as virtualized resource environment, modular infrastructure, automated operation and maintenance management, rapid expansion capability, efficient resource utilization, and reliable redundant backup. While edge computing has advantages in the latency due to the short distance to end users, a variety of business applications have exploded, which has placed higher demands on the basic functions and service performance of the edge servers. This chapter briefly introduces these two trends. Chapter 2 This chapter gives a comprehensive survey of related issues in cloud and edge computing, including (1) the whole cloud computing process, starting from the service access to the data center, to the data transmission control, to the server back- end communication, and to the data synchronization service support, tracking the complete service flow of the data flow and (2) the key two issues in the edge storage and computing. We carry out in-depth related work analysis for each of these topics. Chapter 3 This chapter designs a new type of task access control mechanism, which performs delay prediction on each intermediate server to pre-calculate the overall response delay of the request and redistributes and adjusts the data center resources according to the estimated response delay. The estimated delay of the load stream falls within the user’s patience function. On this basis, this chapter also introduces a delay feedback mechanism to ensure that non-interactive load streams are not greatly affected, ensuring algorithm fairness. Chapter 4 This chapter designs an efficient transmission protocol between the end system and the end user and a dual-aware scheduling mechanism (both task-level awareness and data stream-level awareness) through the end system and improves with the terminal. The ECN mechanism communicates to minimize the completion time of multitasking. Specifically, this chapter first studies the potential impact of the scheduling mechanism considering the task level and data flow level on the data center task scheduling effect, which leads to the task-aware and flow-aware (TAFA) and the band. There are data transfer protocols that improve ECN. Through the task scheduling strategy with priority adjustment, the data stream and task are serialized more reasonably, and the resource competition of the task is minimized. Chapter 5 This chapter addresses the communication issues within the container group for the same application by means of container redistribution between hosts. Specifically, this chapter designs a redistribution mechanism FreeContainer that can sense communication between containers using a novel two-stage online adjustment algorithm. In the first phase, some hosts are vacated to reserve more space for the next-stage redistribution algorithm. The second-stage container redistribution algorithm utilizes an improved variable neighborhood search algorithm to find a
- 8. Preface vii better distribution scheme. The FreeContainer algorithm does not require hardware modifications and is completely transparent to online applications. It was deployed on Baidu’s server cluster and performed extensive measurements and evaluations in a real network environment. The data results of the online application request indicate that FreeContainer can significantly reduce the communication overhead between containers and increase the throughput of the cluster in a traffic burst environment compared with the current adjustment algorithm. Chapter 6 This chapter introduces the BDS system, a transmission scheduling platform for large-scale data synchronization tasks. Specifically, BDS uses a centrally controlled approach to coordinating multiple scheduling tasks. By imple- menting the algorithm, BDS is able to dynamically allocate bandwidth resources between different transport tasks, maximizing traffic based on task priorities. In order to verify the transmission performance of BDS, this chapter has been deployed and verified on Baidu’s intra-domain network and compared with current technology. Chapter 7 This chapter presents AutoSight, a distributed edge caching system for short video network, which significantly boosts cache performance. AutoSight consists of two main components, solving the above two challenges, respectively: (i) the CoStore predictor, which solves the non-stationary and unpredictability of local access pattern by analyzing the complex video correlations, and (ii) a caching engine Viewfinder, which solves the temporal and spatial video popularity problem by automatically adjusting future sights according to video life span. All these inspirations and experiments are based on the real traces of more than 28 million videos with 100 million accesses from 488 servers located in 33 cities. Experiment results show that AutoSight brings significant boosts on distributed edge caching in short video network. Chapter 8 This chapter proposes the computation method of controllability in edge networks by analyzing the controllability of dynamic temporal network. It also calculates the minimum number of driver nodes, so as to ensure the network controllability. These insights are critical for varieties of applications in the future edge network. Beijing, China Yuchao Zhang January 2019 Ke Xu
- 9. Acknowledgments A significant part of this book is the result of work performed in the Computer Science and Technology Department, Tsinghua University. Some parts of the material presented in this book are the work performed in Hong Kong University of Science and Technology (HKUST) and Beijing University of Posts and Telecom- munications (BUPT). We are grateful to all the partners, faculties, and students. Some implementation work was done in Baidu, Huawei, and Kuaishou Companies. We thank them for their efficient cooperation. Also, thanks to the students in my group, namely, Shuang Wu, Pengmiao Li, and Peizhuang Cong, for their help and contributions. Finally, we hope this book will be useful to researchers, students, and other participants working in related topics. It would be great if we can inspire them to make their own contributions in cloud and edge computing field! ix
- 10. Contents 1 Introduction .................................................................. 1 1.1 Research Background................................................... 1 1.2 Content Summary ....................................................... 3 1.3 Key Contributions....................................................... 9 1.4 Chapter Arrangement ................................................... 11 References ..................................................................... 11 2 A Survey of Resource Management in Cloud and Edge Computing... 15 2.1 Latency Optimization for Cloud-Based Service Chains .............. 15 2.2 Toward Shorter Task Completion Time ................................ 16 2.3 Container Placement and Reassignment for Large-Scale Network... 18 2.4 Near-Optimal Network System for Data Replication ................. 20 2.5 Distributed Edge Caching in Short Video Network ................... 22 2.6 The Controllability of Dynamic Temporal Network .................. 23 References ..................................................................... 25 3 A Task Scheduling Scheme in the DC Access Network .................. 33 3.1 Introduction ............................................................. 33 3.2 Dynamic Differentiated Service with Delay-Guarantee............... 35 3.2.1 The Components of Latency ................................... 35 3.2.2 Design Philosophy .............................................. 35 3.2.3 D3G Framework ................................................ 36 3.2.4 Adjusting Resource Allocation................................. 37 3.3 Deployment ............................................................. 37 3.4 D3G Experiment ........................................................ 38 3.4.1 Overall Performance............................................ 38 3.4.2 Algorithm Dynamism .......................................... 39 3.4.3 System Scalability .............................................. 40 3.5 Conclusion .............................................................. 40 References ..................................................................... 41 xi
- 11. xii Contents 4 A Cross-Layer Transport Protocol Design in the Terminal Systems of DC ................................................................ 43 4.1 Introduction ............................................................. 43 4.2 TAFA’s Control Scheme ................................................ 45 4.3 Key Challenges.......................................................... 46 4.4 TAFA: Task-Aware and Flow-Aware................................... 47 4.4.1 Task-Awareness ................................................. 47 4.4.2 Flow-Awareness ................................................ 50 4.4.3 Algorithm Implementation ..................................... 52 4.5 System Stability ......................................................... 53 4.6 TAFA Experiment....................................................... 56 4.6.1 Setup ............................................................ 56 4.6.2 Overall Performance of TAFA ................................. 56 4.6.3 TAFA vs. Task-Aware .......................................... 58 4.6.4 TAFA vs. Flow-Aware .......................................... 59 4.7 Conclusion .............................................................. 62 References ..................................................................... 63 5 Optimization of Container Communication in DC Back-End Servers ........................................................................ 65 5.1 Container Group-Based Architecture .................................. 65 5.2 Problem Definition ...................................................... 66 5.2.1 Objective ........................................................ 66 5.2.2 Constraints ...................................................... 68 5.3 Container Placement Problem .......................................... 70 5.3.1 Problem Analysis ............................................... 70 5.3.2 CA-WFD Algorithm............................................ 71 5.4 Container Reassignment Problem ...................................... 72 5.4.1 Problem Analysis ............................................... 72 5.4.2 Sweep&Search Algorithm...................................... 73 5.5 Implementation.......................................................... 76 5.6 Experiment .............................................................. 79 5.6.1 Performance of CA-WFD ...................................... 80 5.6.2 Performance of Sweep&Search ................................ 83 5.7 Approximation Analysis of Sweep&Search ........................... 85 5.8 Conclusion .............................................................. 87 References ..................................................................... 88 6 The Deployment of Large-Scale Data Synchronization System for Cross-DC Networks...................................................... 91 6.1 Motivation of BDS+ Design ............................................ 91 6.1.1 Baidu’s Inter-DC Multicast Workload ......................... 92 6.1.2 Potentials of Inter-DC Application-Level Overlay ............ 93 6.1.3 Limitations of Existing Solutions .............................. 95 6.1.4 Key Observations ............................................... 97 6.2 System Overview ....................................................... 97
- 12. Contents xiii 6.3 Near-Optimal Application-Level Overlay Network ................... 99 6.3.1 Basic Formulation .............................................. 99 6.3.2 Decoupling Scheduling and Routing .......................... 101 6.3.3 Scheduling ...................................................... 102 6.3.4 Routing .......................................................... 102 6.4 Dynamic Bandwidth Separation........................................ 103 6.4.1 Design Logic .................................................... 104 6.4.2 Integrated to BDS+ ............................................. 104 6.5 System Design .......................................................... 105 6.5.1 Centralized Control of BDS+ .................................. 105 6.5.2 Dynamic Bandwidth Separation of BDS+..................... 107 6.5.3 Fault Tolerance.................................................. 107 6.5.4 Implementation and Deployment .............................. 108 6.6 BDS+ Experiment....................................................... 108 6.6.1 BDS+ Over Existing Solutions................................. 109 6.6.2 Micro-benchmarks.............................................. 110 6.6.3 BDS+’s Dynamic Bandwidth Separation ...................... 114 6.7 Conclusion .............................................................. 117 References ..................................................................... 118 7 Storage Issues in the Edge .................................................. 121 7.1 Introduction ............................................................. 121 7.1.1 The Characteristics of Edge Caching in Short Video Network ......................................................... 122 7.1.2 Limitations of Existing Solutions .............................. 123 7.2 AutoSight Design........................................................ 124 7.2.1 System Overview ............................................... 124 7.2.2 Correlation-Based Predictor: CoStore ......................... 124 7.2.3 Caching Engine: Viewfinder .................................... 125 7.3 AutoSight Experiment .................................................. 126 7.3.1 Experiment Setting ............................................. 126 7.3.2 Performance Comparison ...................................... 126 7.4 Conclusion .............................................................. 127 References ..................................................................... 128 8 Computing Issues in the Edge .............................................. 129 8.1 Background.............................................................. 129 8.2 DND: Driver Node Algorithm.......................................... 131 8.2.1 Parameters and Variable Declarations ......................... 131 8.2.2 Modeling ........................................................ 131 8.2.3 Abstraction of Topology........................................ 133 8.3 DND Experiment........................................................ 134 8.3.1 Communication Radius......................................... 135 8.3.2 Nodes Density .................................................. 135
- 13. xiv Contents 8.3.3 Nodes Velocity .................................................. 136 8.3.4 Control Time .................................................... 137 8.4 Conclusion .............................................................. 137 References ..................................................................... 138
- 14. Chapter 1 Introduction Abstract Data center has many features such as virtualized resource environment, modular infrastructure, automated operation and maintenance management, rapid expansion capability, efficient resource utilization, and reliable redundant backup. While edge computing has advantages in the latency due to the short distance to end users, a variety of business applications have exploded, which has placed higher demands on the basic functions and service performance of the edge servers. This book will analyze these two trends in detail. This chapter introduces the research background, content summary, key contributions, and arrangements of the remaining chapters. 1.1 Research Background The rapid growth of service chains is changing the landscape of cloud-based applications. Different stand-alone components are now handled by cloud servers, providing cost-efficient and reliable services to Internet users. It is known that the workloads from service chains are more complex than the traditional non- interactive (or batch) workloads: for non-interactive workloads, such as scientific computing and image processing, they can be processed on one specific server and do not need interactions from other servers. Being not strictly time-sensitive, they can be scheduled to run anytime as long as the work could finish before a soft deadline. But for the interactive workloads from service chains, they should go through multiple stand-alone components to apply corresponding functions, and this unavoidably introduces additional latency. Meanwhile these interactive chained services typically process real-time user requests, such as business transactional and complex gaming control. Therefore, the performance for service chains is in urgent need to be ensured. Nowadays, data centers have become the cornerstones of modern computing infrastructure and one dominating paradigm in the externalization of IT resources. The data center tasks always consist of rich and complex flows which traverse different parts of the network at potentially different times. To minimize the network contention among different tasks, task serialization was widely suggested. This © Springer Nature Singapore Pte Ltd. 2020 Y. Zhang, K. Xu, Network Management in Cloud and Edge Computing, https://doi.org/10.1007/978-981-15-0138-8_1 1
- 15. 2 1 Introduction approach applies task-level metric and aims to serve one task at a time with synchronized network access. While serialization is a smart design to avoid task- level interference, our study shows that the flow level network contention within a task can however largely affect the task completion time. This prolongs the tail as well as the average task completion time and unavoidably reduces the system applicability to serve delay-sensitive applications. Containerization [1] has become a popular virtualization technology due to many promising properties such as lightweight, scalable, highly portable, and good isolation, and the emergence of software containerization tools, e.g., docker [2], further allows users to create containers easily on top of any infrastructure. Therefore, more and more Internet service providers are deploying their services in the form of containers in modern data centers. For large-scale online service providers, such as Google, Facebook, and Baidu, an important data communication pattern is inter-DC multicast of bulk data – replicating massive amounts of data (e.g., user logs, web search indexes, photo sharing, blog posts) from one DC to multiple DCs in geo-distributed locations. Our study on the workload of Baidu shows that inter-DC multicast already amounts to 91% of inter-DC traffic, which corroborates the traffic pattern of other large-scale online service providers [3, 4]. As more DCs are deployed globally and bulk data are exploding, inter-DC traffic then needs to be replicated in a frequent and efficient manner. While there have been tremendous efforts toward better inter-DC network performance (e.g., [3, 5–9]), the focus has been improving the performance of the wide area network (WAN) path between each pair of DCs. These WAN-centric approaches, however, are incomplete, as they fail to leverage the rich application- level overlay paths across geo-distributed DCs, as well as the capability of servers to store-and-forward data. As illustrated in Fig. 1.1, the performance of inter-DC multicast could be substantially improved by sending data in parallel via multiple overlay servers acting as intermediate points to circumvent slow WAN paths and performance bottlenecks in DC networks. DC A DC B DC C DC B DC C DC A (a) Direct replication over pair-wise inter-DC WANs (b) Replication leveraging overlay paths 1 2 2 2 1 2 Overlay servers Fig. 1.1 A simple network topology illustrating how overlay paths reduce inter-DC multicast completion time. Assume that the WAN link between any two DCs is 1 GB/s, and that A wants to send 3 GB data to B and C. Sending data from A to B and C separately takes 3 s (a), but using overlay paths A → B → C and A → C → B simultaneously takes only 2 s (b). The circled numbers show the order for each data piece is sent
- 16. 1.2 Content Summary 3 It is important to notice that these overlay paths should be bottleneck-disjoint; that is, they do not share common bottleneck links (e.g., A → B → C and A → C → B in Fig. 1.1) and that such bottleneck-disjoint overlay paths are available in abundance in geo-distributed DCs. In recent years, many short video platforms are developing at an incredible speed (such as Kuaishou [10], Douyin/TikTok [11], YouTube Go [12], Instagram Stories [13], and so on), these platforms allow users to record and upload very short videos (usually within 15 s). As a result of the massive short videos uploaded by distributed users, caching problem is becoming more challenging compared with the traditional centralized Content Delivery Network (CDN) whose traffic is dominated by some popular items for a long period of time [14]. To handle scalability and improve users’ Quality of Experience (QoE), the emerging edge computing naturally matches the demand for distributed storage, and the abovementioned platforms thus have resorted to employ edge caching servers to store and deliver the massive short videos, so as to avoid that all requests have to be fetched from the backend/origin server, which usually introduces extra user-perceived latency. In recent years, with the rapid development of Internet of things technology, a variety of smart devices that can be connected to the network have emerged. However, these smart devices are different from the traditionals, especially some of these kinds are highly mobile, such as the rapid driving vehicles in the Internet of Vehicles, which led to the Internet of Vehicles becoming a kind of dynamic networks. In fact, dynamic networks exist in all aspects of our lives, such as delay- tolerant networks, opportunistic-mobility networks, social networks, friendship networks, etc. [15, 16]. The topology of these networks is not static, but changes over time, where the connections among nodes establish or destruct irregularly. This results in rapidly changing network topology, which makes it difficult to hold controllability of the entire network. To address this challenge, it is necessary to propose a way to analyze and solve the controllability of dynamic networks. 1.2 Content Summary In this chapter, we present the measurement and analysis of interactive workloads on Baidu’s cloud platform. The real-world trace indicates that these interactive workloads are suffering from significantly long latency than the general non- interactive workloads. A typical case is shown in Fig. 1.2a where Nuomi is a group buying application, Waimai is a take-out service, and Alipay is an online payment platform. When a user clicks an item on Nuomi, the latency is quite short because this query does not require many interactions among services. However, the story will be different when the user orders a take-out and purchase the item. In this case, the request goes through Nuomi, Waimai, and then Alipay. In other words, this interactive workload consists of several highly dependent operations which have to be processed on different servers separately, like Fig. 1.2b, and there are six procedures for interactive workloads and only two procedures for non-interactive
- 17. 4 1 Introduction (a) (b) Service 1 Service 2 Service 3 User Request User 3 User 1 User 3 Non-interactive Interactive Data flow User 2 nuomi.com waimai.baidu.com alipay.com Fig. 1.2 The processes of interactive and non-interactive workloads workloads. It is easy to see that such interactive workloads in chained applications will introduce extra latency to users because these requests will be handled by different services for multiple times. Unfortunately, we find that most of the existing workload scheduling approaches are aimed to reschedule [17] and leverage different priorities [18, 19] on individual queues. In other words, these optimizations are made on intermediate servers separately, so the overall latency of interactive workloads is still unpredictable. To better optimize the overall latency of chained services, we apply a latency estimation approach to predict overall latency and try to accelerate the interactive workloads. Furthermore, we design a feedback scheme to ensure workload fairness and avoid remarkable degradation of non-interactive workloads. Our real-world deployments on Baidu indicate that the proposed delay-guarantee (D3G) framework can successfully reduce the latency of interactive applications with minimal effect on other workloads. In this chapter, we for the first time investigate the potential to consider both flow level and task level interference together for data center task scheduling. We provide the design of TAFA (task-aware and flow-aware) to obtain better serialization and minimize possible flow and task contentions. TAFA adopts dynamic priority adjustment for the task scheduling. Different from FIFO-LM [20], this design can successfully emulate shortest-task-first scheduling while requires no prior knowledge about the task. Further, TAFA gives a more reasonable and efficient approach to reduce task completion time by considering the relationship among different flows in one task, rather than treating them all the same. With this intelligent adjustment in flow level, TAFA provides the shorter flow waiting time and leading to earlier finish time. As the total waiting time of a task consists of waiting time and processing time, TAFA for the first time combines the two aspects together.
- 18. 1.2 Content Summary 5 To achieve shorter processing time, TAFA solves the dominant resource matching problem and classifies different requirements and VMs into different resource dominant type. By allocating different resource dominant VMs to corresponding resource-dominant requirements, we can achieve more rapid processing time. Generally, each Internet service has several modules which are instantiated as a set of containers, and the containers belonging to the same service often need to communicate with each other to deliver the desired service [21–24], resulting in heavy cross-server communications and downgrading service performance [19, 21]. If these containers are placed on the same server, the communication cost can be greatly reduced. However, the containers belonging to the same service are generally intensive to the same resource (e.g., containers of the big data analytics services [25–27] are usually CPU-intensive, and containers of the data transfer applications [4, 28–31] are usually network I/O-intensive). Assigning these contain- ers on the same server may cause heavily imbalanced resource utilization of servers, which could affect the system availability, response time, and throughput [32, 33]. First, it prevents any single server from getting overloaded or breaking down, which improves service availability. Second, servers usually generate exponential response time when the resource utilization is high [34]; load balancing guarantees acceptable resource utilizations for servers, so that the servers can have fast response time. Third, no server will be a bottleneck under balanced workload, which improves the overall throughput of the system. Figure 1.3 shows an example. Suppose there are two services (denoted by SA and SB) to be deployed on two servers. Each service has two containers (CA1, CA2, and CB1, CB2, respectively). The containers of SA are CPU-intensive, while the containers of SB are network I/O-intensive. Figure 1.3a shows a solution which assigns one of SA’s containers and one of SB’s containers on each server. This approach achieves high resource utilization on both CPU and network I/O, but incurs high communication cost between the two servers. Figure 1.3b shows another solution where the containers of the same service are assigned on the same server. The communication overhead is thus significantly reduced; however, the utilization of CPU and network I/O is highly imbalanced on the two servers. Server CPU I/O CA2 CA2 CA1 Server CPU I/O CB2 CB1 CB2 CB1 Network CA1 (a) Server CPU I/O CA2 CA1 CA2 CA1 Server CPU I/O CB2 CB1 CB2 CB1 Network (b) Fig. 1.3 The conflict between container communication and server resource utilization. (a) High network communication overhead, balanced resource utilization. (b) Imbalanced resource utilization, low network communication overhead
- 19. 6 1 Introduction Table 1.1 Resource utilization in a data center from Baidu (http://www. baidu.com) Resource Top 1% Top 5% Top 10% Mean CPU 0.943 0.865 0.821 0.552 MEM 0.979 0.928 0.890 0.626 SSD 0.961 0.927 0.875 0.530 We further explore the conflict between container communication and resource utilization in a data center with 5,876 servers from Baidu. According to our knowledge, the containers of the same service in this data center are placed as close as possible in order to reduce the communication cost. Table 1.1 gives the top 1%, top 5% and top 10% CPU, MEM (memory) and SSD (solid-state Drives) utilization of servers in this data center, which shows that the utilization of resources is highly imbalanced among servers. Reducing container communication cost while keeping balanced server resource utilization is never an easy problem. In this chapter, we try to address such conflict in large-scale data centers. Specifically, such conflict lies in two related phases of an Internet service’s life cycle, i.e., container placement and container reassignment, and we accordingly study two problems. The first is container placement problem, which strives to place a set of newly instantiated containers into a data center. The objective of this phase is to balance resource utilization while minimizing the communication cost of these containers after placement. The second is container reassignment problem, which tries to optimize a given placement of containers by migrating containers among servers. Such reassignment approach can be used for online periodical adjustment of the placement of containers in a data center. We formulate these two problems as multi-objective optimization problems, which are both NP hard. For the container placement problem, we propose an efficient Communication Aware Worst Fit Decreasing (CA-WFD) algorithm, which subtly extends the classical Worst Fit Decreasing bin packing algorithm to container placement. For the container reassignment problem, we propose a two-stage algorithm named Sweep&Search which can seek a container migration plan efficiently. We deploy our algorithms in Baidu’s data centers and conduct extensive experiments to evaluate the performance. The results show that the proposed algorithms can effectively reduce the communication cost while simultaneously balancing the resource utilization among servers in real systems. The results also show that our algorithms outperform the state-of-the-art strategies up to 90% used by some top containerization service providers. This chapter first introduces BDS+, an application-level centralized near-optimal network system, which splits data into fine-grained units, and sends them in parallel via bottleneck-disjoint overlay paths with dynamic bandwidth sharing. These paths are selected dynamically in response to changes in network conditions and the data delivery status of each server. Note that BDS+ selects application- level overlay paths and is therefore complementary to network-layer optimization of WAN performance. While application-level multicast overlays have been applied
- 20. 1.2 Content Summary 7 in other contexts (e.g., [35–38]), building one for inter-DC multicast traffic poses two challenges. First, as each DC has tens of thousands of servers, the resulting large number of possible overlay paths makes it unwieldy to update overlay routing decisions at scale in real time. Prior work either relies on local reactive decisions by individual servers [39–41], which leads to suboptimal decisions for lack of global information, or restricts itself to strictly structured (e.g., layered) topologies [42], which fails to leverage all possible overlay paths. Second, even a small increase in the delay of latency-sensitive traffic can cause significant revenue loss [43], so the bandwidth usage of inter-DC bulk-data multicasts must be tightly controlled to avoid negative impact on other latency-sensitive traffic. To address these challenges, BDS+ fully centralizes the scheduling and routing of inter-DC multicast. Contrary to the intuition that servers must retain certain local decision-making to achieve desirable scalability and responsiveness to net- work dynamics, BDS+’s centralized design is built on two empirical observations (Sect. 6.2): (1) while it is hard to make centralized decisions in real time, most multicast data transfers last for at least tens of seconds and thus can tolerate slightly delayed decisions in exchange for near-optimal routing and scheduling based on a global view; (2) centrally coordinated sending rate allocation is amenable to minimizing the interference between inter-DC multicast traffic and latency-sensitive traffic. The key to making BDS+ practical is how to update the overlay network in near real-time (within a few seconds) in response to performance churns and dynamic arrivals of requests. BDS+ achieves this by decoupling its centralized control into two optimization problems, scheduling of data transfers and overlay routing of individual data transfers. Such decoupling attains provable optimality and, at the same time, allows BDS+ to update overlay network routing and scheduling in a fraction of second; this is four orders of magnitude faster than solving routing and scheduling jointly when considering the workload of a large online service provider (e.g., sending 105 data blocks simultaneously along 104 disjoint overlay paths). In practice, there is always a fixed upper bound of available bandwidth for bulk-data multicast, because multicast overlay network shares the same inter-DC WAN with online latency-sensitive traffic. Existing solutions always reserve a fixed amount of bandwidth for the latency-sensitive traffic, according to its peak value. This guarantees the strict bandwidth separation, but the side effect is the waste of bandwidth, especially when the online traffic is in its valley. To further improve link utilization, BDS+ implements dynamic bandwidth separation that can predict online traffic and reschedule bulk-data transfer. In other words, BDS+ achieves dynamic bandwidth separation between bulk-data multicast and online traffic to further speed up data transfer. We have implemented a prototype and integrated it in Baidu. We first deployed BDS+ in 10 DCs and ran a pilot study on 500 TB of data transfer for 7 days (about 71 TB per day). Our real-world experiments show that BDS+ achieves 3– 5× speedup over Baidu’s existing solution named Gingko, and it can eliminate the incidents of excessive bandwidth consumption by bulk-data transfers. Using micro-benchmarking, we show that BDS+ outperforms techniques widely used in
- 21. 8 1 Introduction CDNs, BDS+ can handle the workload of Baidu’s inter-DC multicast traffic with one general-purpose server, and BDS+ can handle various failure scenarios.1 We then use trace-driven simulations to evaluate BDS+ with dynamic bandwidth separation; the results show that BDS+ further speeds up the bulk data transfer by 1.2 to 1.3 times in the network where online and offline services are mixed deployed. There have been tremendous efforts toward better caching performance in traditional CDN, and these caching algorithms can be classified into two categories: the simple but effective reactive caching algorithms, such as First-In First-Out (FIFO), Least Recently Used (LRU), Least Frequently Used (LFU), k-LRU, and their variants, and the proactive caching algorithms, such as DeepCache [44]. These prior solutions work well in traditional centralize-controlled CDN, but become invalid in the emerging short video network due to the following two essential differences. First, the non-stationary user video access pattern. The basic assumption of reactive caching policies is the stationary user access pattern, i.e., recently requested or frequently requested content in the past should be kept in cache because these policies assume that such contents have greater chance of being visited in the future. While a study in [14] shows that in short video network, popular content will become expired very quickly (within tens of minutes), indicating that the popularity in the past could not represent that in the future, and this is the root cause of the failure of these reactive policies (see Sect. 7.1.1). Second, The temporal and spatial video popularity pattern, i.e., the change of video popularity, varies in different edge caching servers during different time periods. Our study on the workload of Kuaishou shows that it takes less than 1 h for a popular video to become unpopular during peak hours while takes more than 3 h during late midnight. Existing proactive caching policies that try to predict future content popularity always focus on a fixed sight, making them fail in edge caching scenarios. To address the above challenges, this chapter presents AutoSight, a distributed caching mechanism that works in edge caching servers for short video network. AutoSight allows edge servers to retain respective local caching sight to adapt to local video access pattern and video life span. AutoSight’s distributed design is built on two empirical observations: (1) although the historical video access data is non-stationary on individual edge server (see Fig. 7.1), making it difficult to make popularity prediction, there is sufficient correlations among videos within the same edge server, because users tend to request related videos, contributing much cross visits in edge servers, and thus improves distributed predictions, and (2) temporal and spatial video popularity pattern brings challenge for future sight of caching policy (see Fig. 7.2), but distributed design allows adaptive future sights, enabling edge servers to make decisions according to different video expiration speeds. 1As the existing solutions are with fixed bandwidth separation, so in these series of experiments, we use BDS+ without dynamic bandwidth separation as comparation, while BDS+ with dynamic bandwidth separation is evaluated separately.
- 22. 1.3 Key Contributions 9 We have implemented a prototype of AutoSight and evaluate it with Kuaishou data traces. Experiments show that AutoSight achieves much higher hit rate com- pared with both reactive caching policies and the state-of-the-art proactive caching policies. There has been some research and progress on controllability of networks in recent years. However, these research just proved the controllability of complex networks in mathematical theory, but do not apply the theory to the controllability of mobile dynamic networks. This chapter refers to the mathematical theory knowledge of these papers and applies it to the controllability of dynamic networks. For a dynamic network, it needs several driver nodes to ensure the controllability of the entire network. By analyzing the connection state of the dynamic network during control time, the minimum number of driver nodes can be obtained. 1.3 Key Contributions The main contributions are summarized as follows: • We present the measurement and latency analysis of service chains in Baidu networks and disclose the long latency for interactive workloads. • We design the D3G algorithm to accelerate interactive workloads in a global manner other than in each independent server and leverage a latency estimation algorithm and a feedback scheme to ensure fairness. • We evaluate our methods on servers in Baidu networks, and the extensive experiment results show that D3G succeeds in accelerating interactive chained applications while ensuring workload fairness. In short, this chapter mainly makes three contributions, which are described as follows. Firstly, we point out that the flow contentions in one task will also make system performance degrade, and a task-aware scheme which ignores flow relationship will achieve longer completion time. We give a simple example in Sect. 4.1 and analyze the disadvantages of leaving out information on flow contention. Secondly, we design an scheduling framework called TAFA, which can achieve both task-awareness and flow-awareness. Task-awareness ensures short tasks are prioritized over long tasks and enables TAFA to emulate STF scheduling without knowing the task size beforehand. Flow-awareness optimizes scheduling order, and achieves shorter task completion time. Thirdly, we solved the practical issue in the designing framework. We make TAFA applicable by considering realistic environment which is in multiple resources and heterogeneous VMs. To avoid mismatching between resource requirements and allocation, we solve the dominant resource matching problem, which will occur when heterogenous requirements of flows conflict with heterogenous virtual machine (VM) configurations. By using the concept of dominant resource, we can choose the proper VM for particular flows.
- 23. 10 1 Introduction This chapter is extended from our preliminary work [45] with significant improvements including: • We disclose a new problem (i.e., the container placement problem) that places a set of newly instantiated containers into a data center, which is a necessary and important phase in services’ life cycle. • We propose the CA-WFD algorithm to solve the container placement problem and conduct extensive experiments to evaluate the performance. • We refine the algorithms proposed for the container reassignment problem and significantly extend the experimental study for this problem. Our contributions are summarized as follows: • Characterizing Baidu’s workload of inter-DC bulk-data multicast to motivate the need of application-level multicast overlay networks (Sect. 6.1). • Presenting BDS+, an application-level multicast overlay network that achieves near-optimal flow completion time by a centralized control architecture (Sects. 6.2 and 6.3). • Making dynamic bandwidth separation to further improve link utilization in the network where online and offline services are mixed deployed (Sects. 6.2 and 6.4). • Demonstrating the practical benefits of BDS+ by a real-world pilot deployment and large-scale simulations in Baidu (Sects. 6.5 and 6.6). Our contributions are summarized as follows: • Characterizing Kuaishou’s workload of short video network to motivate the need of distributed edge caching policy. • Presenting AutoSight, a distributed edge caching mechanism working in edge caching servers for short video network, which solves the problem of non- stationary user access pattern and temporal/spatial video popularity pattern. • Demonstrating the practical benefits of AutoSight by real traces. First, this chapter raised some issues of controllability challenges faced by dynamic networks; then took the vehicle network on a road as an example with the moving speed, communication radius, average density of vehicles, and the control time as different variable parameters; and designed an algorithm to calculate the optimal solution of the minimum number of driver nodes required for a dynamic network. Finally, after the experiment of simulation data, we got the influence of the vehicle’s moving speed, communication radius, average density, and the control time on the number of required driver nodes of a vehicle network. By deploying a minimum of driver nodes to ensure the controllability of the entire network, and maximizes resource conservation, improves efficiency, ultimately, guides the deployment of mobile networks.
- 24. References 11 1.4 Chapter Arrangement Chapter 3 The rest of this chapter is organized as follows. We measure the performance and latency of different service workloads in Sect. 3.1. To reduce total latency for interactive workloads, we design a new algorithm called D3G and present the detailed design in Sect. 3.2. Section 3.3 describes the deployment of D3G, and Sect. 3.4 evaluates the experiment results on Baidu networks. Section 3.5 concludes this chapter. Chapter 4 The rest of this chapter is organized as follows. We introduce the main system framework TAFA in Sect. 4.4. Section 4.5 proves the properties of TAFA, and Sect. 4.6 shows the simulation results. Finally, we conclude this chapter and point out the future work in Sect. 4.7. Chapter 5 The rest of this chapter is structured as follows. Section 5.1 intro- duces the architecture of container group-based services. Definitions of container placement problem and container reassignment problem are given in Sect. 5.2. Our solutions to the two problems are proposed in Sects. 5.3 and 5.4, respectively. Section 5.6 compares our solutions with state-of-the-art designs by extensive evaluations. We implement our solutions in large-scale data centers of Baidu, and the details are given in Sect. 5.5. At last, Sect. 5.8 concludes the chapter. Chapter 6 The rest of this chapter is organized as follows. We provided a case for an application-level multicast overlay network in Sect. 6.1. To optimize inter- DC multicasts on overlay network with dynamical separation with latency-sensitive traffic, we present BDS+, a fully centralized near-optimal network system with dynamic bandwidth separation for data inter-DC multicast in Sect. 6.2. Section 6.5 presents the system design and implemetation of BDS+, and Sect. 6.6 compared BDS+ with three existing solutions. Section 6.7 concludes this chapter. Chapter 7 The rest of this chapter is organized as follows. We characterized the short video network and illustrated the essential difference with traditional CDN in Sect. 7.1. We show the AutoSight design in Sect. 7.2, and we evaluate our approach AutoSight using real traces in Sect. 7.3. Section 7.4 concludes this chapter. Chapter 8 The remainder of this chapter is organized as follows. In Sect. 8.1 we introduce the background of the application scenario. Section 8.2 describes some variables and specific formulas of our modeling. Section 8.3 presents the results and analysis. Finally, the conclusions and future work are laid out in Sect. 8.4. References 1. Soltesz, S., Fiuczynski, M.E., Bavier, A., Peterson, L.: Container-based operating system vir- tualization: a scalable, high-performance alternative to hypervisors. In: ACM Sigops/Eurosys European Conference on Computer Systems, pp. 275–287 (2007) 2. Docker: http://www.docker.com/ (2016)
- 25. 12 1 Introduction 3. Kumar, A., Jain, S., Naik, U., Raghuraman, A., Kasinadhuni, N., Zermeno, E.C., Gunn, C.S., Ai, J., Carlin, B., Amarandei-Stavila, M., et al.: BwE: flexible, hierarchical bandwidth allocation for WAN distributed computing. In: ACM SIGCOMM, pp. 1–14 (2015) 4. Zhang, Y., Xu, K., Yao, G., Zhang, M., Nie, X.: Piebridge: a cross-dr scale large data transmission scheduling system. In: Proceedings of the 2016 Conference on ACM SIGCOMM 2016 Conference, pp. 553–554. ACM (2016) 5. Savage, S., Collins, A., Hoffman, E., Snell, J., Anderson, T.: The end-to-end effects of Internet path selection. ACM SIGCOMM 29(4), 289–299 (1999) 6. Jain, S., Kumar, A., Mandal, S., Ong, J., Poutievski, L., Singh, A., Venkata, S., Wanderer, J., Zhou, J., Zhu, M., et al.: B4: experience with a globally-deployed software defined WAN. ACM SIGCOMM 43(4), 3–14 (2013) 7. Hong, C.-Y., Kandula, S., Mahajan, R., Zhang, M., Gill, V., Nanduri, M., Wattenhofer, R.: Achieving high utilization with software-driven WAN. In: ACM SIGCOMM, pp. 15–26 (2013) 8. Zhang, H., Chen, K., Bai, W., Han, D., Tian, C., Wang, H., Guan, H., Zhang, M.: Guaranteeing deadlines for inter-datacenter transfers. In: EuroSys, p. 20. ACM (2015) 9. Zhang, Y., Xu, K., Wang, H., Li, Q., Li, T., Cao, X.: Going fast and fair: latency optimization for cloud-based service chains. IEEE Netw. 32, 138–143 (2017) 10. kuaishou: Kuaishou. https://www.kuaishou.com (2019) 11. TikTok: Tiktok. https://www.tiktok.com (2019) 12. Go, Y.: Youtube go. https://youtubego.com (2019) 13. Stories, I.: Instagram stories. https://storiesig.com (2019) 14. Zhang, Y., Li, P., Zhang, Z., Bai, B., Zhang, G., Wang, W., Lian, B.: Challenges and chances for the emerging shortvideo network. In: Infocom, pp. 1–2. IEEE (2019) 15. Casteigts, A., Flocchini, P., Quattrociocchi, W., Santoro, N.: Time-varying graphs and dynamic networks. Int. J. Parallel Emergent Distrib. Syst. 27(5), 387–408 (2012) 16. Xiao, Z., Moore, C., Newman, M.E.J.: Random graph models for dynamic networks. Eur. Phys. J. B 90(10), 200 (2016) 17. Alizadeh, M., Yang, S., Sharif, M., Katti, S., McKeown, N., Prabhakar, B., Shenker, S.: pFabric: minimal near-optimal datacenter transport. ACM SIGCOMM Comput. Commun. Rev. 43(4), 435–446 (2013). ACM 18. Dogar, F.R., Karagiannis, T., Ballani, H., Rowstron, A.: Decentralized task-aware scheduling for data center networks. ACM SIGCOMM Comput. Commun. Rev. 44(4), 431–442 (2014). ACM 19. Zhang, Y., Xu, K., Wang, H., Shen, M.: Towards shorter task completion time in datacenter networks. In: 2015 IEEE 34th International Performance Computing and Communications Conference (IPCCC), pp. 1–8. IEEE (2015) 20. Dogar, F.R., Karagiannis, T., Ballani, H., Rowstron, A.: Decentralized task-aware scheduling for data center networks. SIGCOMM Comput. Commun. Rev. 44, 431–442 (2014) 21. Yu, T., Noghabi, S.A., Raindel, S., Liu, H., Padhye, J., Sekar, V.: Freeflow: high performance container networking. In: Proceedings of the 15th ACM Workshop on Hot Topics in Networks, pp. 43–49. ACM (2016) 22. Burns, B., Oppenheimer, D.: Design patterns for container-based distributed systems. In: 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 16) (2016) 23. Zhang, Y., Xu, K., Wang, H., Li, Q., Li, T., Cao, X.: Going fast and fair: latency optimization for cloud-based service chains. IEEE Netw. 32(2), 138–143 (2018) 24. Shen, M., Ma, B., Zhu, L., Mijumbi, R., Du, X., Hu, J.: Cloud-based approximate constrained shortest distance queries over encrypted graphs with privacy protection. IEEE Trans. Inf. Forensics Secur. 13(4), 940–953 (2018) 25. Ananthanarayanan, G., Kandula, S., Greenberg, A.G., Stoica, I., Lu, Y., Saha, B., Harris, E.: Reining in the outliers in map-reduce clusters using mantri. OSDI 10(1), 24 (2010) 26. Li, M., Andersen, D.G., Park, J.W., Smola, A.J., Ahmed, A., Josifovski, V., Long, J., Shekita, E.J., Su, B.-Y.: Scaling distributed machine learning with the parameter server. In: 11th
- 26. References 13 USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pp. 583– 598 (2014) 27. Wu, X., Zhu, X., Wu, G.-Q., Ding, W.: Data mining with big data. IEEE Trans. Knowl. Data Eng. 26(1), 97–107 (2014) 28. Zhang, Y., Jiang, J., Xu, K., Nie, X., Reed, M.J., Wang, H., Yao, G., Zhang, M., Chen, K.: Bds: a centralized near-optimal overlay network for inter-datacenter data replication. In: Proceedings of the Thirteenth EuroSys Conference, p. 10. ACM (2018) 29. Xu, K., Li, T., Wang, H., Li, H., Wei, Z., Liu, J., Lin, S.: Modeling, analysis, and implemen- tation of universal acceleration platform across online video sharing sites. IEEE Trans. Serv. Comput. 11, 534–548 (2016) 30. Wang, H., Li, T., Shea, R., Ma, X., Wang, F., Liu, J., Xu, K.: Toward cloud-based distributed interactive applications: measurement, modeling, and analysis. IEEE/ACM Trans. Netw. 26(99), 1–14 (2017) 31. Zhang, Y., Xu, K., Shi, X., Wang, H., Liu, J., Wang, Y.: Design, modeling, and analysis of online combinatorial double auction for mobile cloud computing markets. Int. J. Commun. Syst. 31(7), e3460 (2018) 32. Gavranović, H., Buljubašić, M.: An efficient local search with noising strategy for Google machine reassignment problem. Ann. Oper. Res. 242, 1–13 (2014) 33. Wang, T., Xu, H., Liu, F.: Multi-resource load balancing for virtual network functions. In: IEEE International Conference on Distributed Computing Systems (2017) 34. Hong, Y.-J., Thottethodi, M.: Understanding and mitigating the impact of load imbalance in the memory caching tier. In: Proceedings of the 4th Annual Symposium on Cloud Computing, p. 13. ACM (2013) 35. Liebeherr, J., Nahas, M., Si, W.: Application-layer multicasting with Delaunay triangulation overlays. IEEE JSAC 200(8), 1472–1488 (2002) 36. Wang, F., Xiong, Y., Liu, J.: mTreebone: a hybrid tree/mesh overlay for application-layer live video multicast. In: ICDCS, p. 49 (2007) 37. Andreev, K., Maggs, B.M., Meyerson, A., Sitaraman, R.K.: Designing overlay multicast networks for streaming. In: SPAA, pp. 149–158 (2013) 38. Mokhtarian, K., Jacobsen, H.A.: Minimum-delay multicast algorithms for mesh overlays. IEEE/ACM TON, 23(3), 973–986 (2015) 39. Kostić, D., Rodriguez, A., Albrecht, J., Vahdat, A.: Bullet: high bandwidth data dissemination using an overlay mesh. ACM SOSP 37(5), 282–297 (2003). ACM 40. Repantis, T., Smith, S., Smith, S., Wein, J.: Scaling a monitoring infrastructure for the Akamai network. ACM Sigops Operat. Syst. Rev. 44(3), 20–26 (2010) 41. Huang, T.Y., Johari, R., Mckeown, N., Trunnell, M., Watson, M.: A buffer-based approach to rate adaptation: evidence from a large video streaming service. In: SIGCOMM, pp. 187–198 (2014) 42. Nygren, E., Sitaraman, R.K., Sun, J.: The Akamai network: a platform for high-performance internet applications. ACM SIGOPS Oper. Syst. Rev. 44(3) (2010) 43. Zhang, Y., Li, Y., Xu, K., Wang, D., Li, M., Cao, X., Liang, Q.: A communication-aware container re-distribution approach for high performance VNFs. In: IEEE ICDCS 2017, pp. 1555–1564. IEEE (2017) 44. Narayanan, A., Verma, S., Ramadan, E., Babaie, P., Zhang, Z.-L.: Deepcache: a deep learning based framework for content caching. In: Proceedings of the 2018 Workshop on Network Meets AI & ML, pp. 48–53. ACM (2018) 45. Zhang, Y., Li, Y., Xu, K., Wang, D., Li, M., Cao, X., Liang, Q.: A communication- aware container re-distribution approach for high performance VNFs. In: IEEE International Conference on Distributed Computing Systems (2017)
- 27. Chapter 2 A Survey of Resource Management in Cloud and Edge Computing Abstract This chapter is to summarize the processing of the business, starting from the service access to the data center, to the data transmission control, to the server back-end communication, and the data synchronization service support, tracking the complete service flow of the data flow. And carry out comprehensive and in-depth research work for each of these links. 2.1 Latency Optimization for Cloud-Based Service Chains As more applications are deployed on clouds for better system scalability and lower operation cost, service chains are developing quickly. Many studies have shown that the latency is particularly problematic when interaction latency occurs together with network delays [1]. To minimize the latency on cloud-based applications, many researchers focused on minimizing datacenter latency, which does advance the state of the art [2]. We can classify these literatures into two categories. First, some literatures focused on network and processing latency. For example, Webb et al. [3] proposed a nearest server assignment to reduce the client-server latency. Vik explored the spanning tree problems in a distributed interactive application system for latency reduction in [4]. Moreover, [5] and [6] introduced a game theory into this topic and modeled the latency problem in DC as a bargaining game, and Seung guaranteed resource allocation by proposing resource allocation [7]. The other kind of related work is web service, which is an application model for decentralized computing and an effective mechanism for data and service integration on the Web [8]. Web service is becoming relatively mature recently. Some studies succeeded in dissecting latency contributors [9], showing that back-office traffic accounts for a significant fraction of web transactions in terms of requests and responses [10]. Although the above studies have already made excellent latency optimization, they ignored the interactions among multiple services (e.g., the case in Fig. 1.2), and the interactive workloads are suffering from longer latencies due to the processing procedures on multiple intermediate servers. Many researchers investigated the latency for interactive applications, and their researches showed that although these © Springer Nature Singapore Pte Ltd. 2020 Y. Zhang, K. Xu, Network Management in Cloud and Edge Computing, https://doi.org/10.1007/978-981-15-0138-8_2 15
- 28. 16 2 A Survey of Resource Management in Cloud and Edge Computing applications are quite delay-sensitive, service performance is greatly affected by interactions. To address this problem, some studies suggested that the interactions of different services should be further dissected to better understand the performance implications [10], and some researchers have already began to pay attention to interaction latency [11]. Our study explores the potential to reduce the response time for service chains and guarantee the non-interactive workloads simultaneously. In particular, we accelerate the interactive workloads by building a new dedicated queue and try to adjust resource allocation among different queues. By leveraging a feedback scheme, we can bound the influence on non-interactive workloads. We’ll describe the algorithm in Sect. 3.2 in detail after introducing our motivation in Sect. 3.1. 2.2 Toward Shorter Task Completion Time Although the ever-increasing used data center networks have configured high bandwidth and calculative ability, the task completion time still can be reduced to a large extent [12]. In this chapter, we describe the nature of today’s datacenter transport protocols, either flow-aware or task-aware and how the awareness of different levels does isolate with each other. As a result, although flow completion time or task completion time seems to reduce obviously, flow-aware protocols are blind to task level and vice versa. In particular, good flow-level awareness can help make task completion time shorter, while good task-level awareness can help flows cooperate harmonically. Figure 2.1 shows the development history of scheduling protocols, from DCTCP (2010) to FIFO-LM (2014), which can be categorized into two broad classes, flow- aware and task-aware, both of which do advance the state-of-the-art technique. We’ll give a brief description and explanation to this progress. As the founder of many flow-aware TCP-like protocols, DCTCP [13] leverages Explicit Congestion Notification (ECN) in the network to provide feedback to end hosts. Experiments show that DCTCP delivers better throughput than TCP while 2010 2011 2012 2013 2014 2015 Time Flow Aware Task Aware Task&Flow Aware DCTCP D3 PDQ D2TCP pFabric pase FIFO-LM TAFA Fig. 2.1 Brief development history of transport protocols
- 29. 2.2 Toward Shorter Task Completion Time 17 using 90% less buffer because it elegantly reduces the queue length. However, it is a deadline-agnostic protocol that equally throttles all flows, irrespective of whether their deadlines are near or far, so it may be less effective for the online data- intensive (OLDI) applications [14]. Motivated by these observations, D3 [15] use explicit rate control for the datacenter environment according to flow deadlines. D3 can determine the rate needed to satisfy the flow deadline when having the knowledge of flow’s sizes and deadlines. Although it outperforms TCP in terms of short flow latency and burst tolerance, D3 has the practical drawbacks of requiring changes to the switch hardware, which makes it not able to coexist with legacy TCP [14]. Deadline-Aware Datacenter TCP (D2TCP) [14] is a deployable transport protocol compared to D3. Via a gamma correction function, D2TCP uses ECN feedback and deadlines to modulate the congestion window. Besides, D2TCP can coexist with TCP without hurting bandwidth or deadline. Preemptive Distributed Quick (PDQ) flow scheduling [16] is designed to complete flows quickly and meet flow deadlines, and it builds on traditional real-time scheduling techniques: earliest deadline first and shortest job first, which help PDQ outperform TCP, RCP [17], and D3 significantly. pFabric [18] decouples flow scheduling from rate control. Unlike the protocols above, in pFabric, each flow carries a single priority number set independently, according to which switches execute a scheduling/dropping mechanism. Although pFabric achieves near-optimal flow completion time, it does not support work conservation in a multi-hop setting because end-hosts always send at maximum rate. To make these flows back off and let a lower priority flow at a subsequent hop, we need an explicit feedback from switches, i.e., a higher layer control. From a network perspective, tasks in DCNs typically comprise multiple flows, which traverse different servers at different times. Treating flows of one task in isolation will make flow-level optimizing while hurting task completion time. To solve this boundedness in flow-aware schemes, task-aware protocols have been proposed to explicitly take the higher layer information into consideration. A task-aware scheduling was proposed by Fahad R. Dogar in [12]. Using First- In-First-Out scheme to reduce both of the average and the tail task completion time, Dogar implemented First-In-First-Out with Limited Multiplexing (FIFO-LM) to change the level of multiplexing when heavy tasks are encountered, which can help heavy tasks not being blocked, or even starved. But as we all know, FIFO is not the most effective method to reduce average completion time no matter in flow level or task level, and the simple distinguish just between elephant tasks and mouse tasks is in coarse granularity as [19] said that the DCNs should be more load, more differentiation. Further, [20] and [21] give methods that can ensure user-level performance guarantee. Without cross-layer cooperation, these protocols have great blindness to each other, making scheduling inefficient. In order to give our method the advantages from both flow level and task level, we praise TAFA with the idea of co-assist that making flow-level schedule help task complete early and task-level schedule help flows mutually related. Our work performs well even in multiple resource-sharing environment.
- 30. 18 2 A Survey of Resource Management in Cloud and Edge Computing 2.3 Container Placement and Reassignment for Large-Scale Network In this section, we survey some problems that are related to our problem, including multi-resource generalized assignment problem, Google Machine Reassignment Problem, traffic-aware virtual machine placement, network function placement, and container deployment and migration. Multi-resource Generalized Assignment Problem (MRGAP) MRGAP [22, 23] is an extension of the Generalized Assignment Problem (GAP) [24, 25], where multiple resources are associated with the items and bins. Solutions to MRGAP usually contain two phases. The first phase aims to obtain an initial feasible solution, and the second phase attempts to further improve the solution. Gavish et al. [23] proposed two heuristics to generate the initial solution and a branch and bound algorithm to improve the solution. Privault et al. [26] computed the initial solution by the bounded variable simplex method and optimized the solution by a simulated annealing algorithm. Mitrović-Minić and Punnen [27] and Yagiura et al. [28] generated a random initial solution in the first phase and adopted local searching techniques in the second phase. Mazzola and Wilcox [29] combined Pivot and Complement (P&C) and the heuristic proposed in [23] to obtain high-quality solutions. In [30], Shtub et al. proposed a gradient descent-based solution to the dynamic MRGAP (DMRGAP), where the resource requirements of items change over time and an item can be assigned to several bins. Although we show that MRGAP is equivalent to the simplified CPP in Sect. 5.3.1, we emphasize that CPP and CRP are more complex than MRGAP because of the containerization-specific constraints (i.e., conflict, spread, co-locate, and transient constraints), which makes above solutions inapplicable in our scenarios. Google Machine Reassignment Problem (GMRP) GMRP was formulated by the Google research team as a subject of ROADEF/EURO Challenge, which aims to maximize the resource usage by reassigning processes among the machines in data centers. Gavranović et al. proposed the winner solution [31] noisy local search (NLS), which combines local searching techniques and noising strategy in reallocation. Different from NLS, we depart the reassignment into two steps, namely, Sweep and Search. With the help of Sweep, we mitigate the hot hosts and obtain better initial conditions for the following local searching procedure. The evaluation result in Sect. 5.6 shows that Sweep&Search yields significantly better results than directly applying local searching techniques. Traffic-Aware Virtual Machine Placement Like containerization, virtual machine (VM) is also a popular virtualization technique, where isolated operation systems run above a hypervisor layer on bare metals. Since each VM runs a full operating system [32], VMs usually have bigger sizes and consume more power than containers. Hence, traditional VM placement mainly concerns about optimization of energy consumption, resource utilizations, and VM migration overhead [33]. Since the pioneer work of [34], many efforts have been made to mitigate inter-server
- 31. 2.3 Container Placement and Reassignment for Large-Scale Network 19 communications by traffic-aware VM placement [35–44]. Meng et al. [34] defined the traffic-aware VM placement problem and proposed a two-tier approximate algorithm to minimize inter-VM communications. Choreo [36] adopts a greedy heuristic to place VMs to minimize application completion time. Li et al. [37] proposed a series of traffic-aware VM placement algorithms to optimize traffic cost as well as single-dimensional resource utilization cost. Rui et al. [42] adopt a system optimization method to re-optimize VM distributions for joint optimization of resource load balancing and VM migration cost. Different from this work, we optimize both communication overhead and multi-resource load balancing. Besides, since containers can be deployed in VMs instead of physical machines, the solutions proposed in our book are orthogonal to these VM placement strategies. Therefore, VM resource utilization and inter-VM communications could be optimized by container placement/reassignment, and that of physical machines could be optimized by VM placement. Network Function Placement Network functions virtualization (NFV) has recently gained wide attention from both industry and academia, making the study of their placement a popular research topic [45–60]. Wang et al. [45] studied the flow-level multi-resource load balancing problem in NFV and proposed a distributed solution based on the proximal Jacobian ADMM (alternating direction method of multipliers). Marotta et al. [49] proposed a mathematical model based on the robust optimization theory to minimize the power consumption of the NFV infrastructure. In [53], an affinity-based heuristic is proposed to minimize inter-cloud traffic and response time. Zhang et al. [54] proposed a Best-Fit Decreasing-based heuristic algorithm to place network functions to achieve high utilization of single dimensional sources. Taleb et al. studied the network function placement problem from many aspects, including minimizing path between users and their respective data anchor gateways [55], measuring existing NFV placement algorithms [56], placing Packet Data Network (PDN) Gateway network functionality and Evolved Packet Core (EPC) in the cloud [57, 58, 60], and modeling cross-domain network slices for 5G [59]. Since network functions work in chains and containers are deployed by groups, the communication patterns are totally different between the two systems. Hence, the communication optimization solutions in NFV are non- applicable to container placement. Besides, none of these works aim at the joint optimization of communication overhead and multi-resource load balancing in data centers. Container Deployment and Migration A lot of work has been studied to deploy containers among virtual machines or physical machines for various optimization purposes. Zhang et al. [61] proposed a novel container placement strategy for improving physical resource utilization. The works [62–64] studied the container placement problem for minimizing energy consumption in the cloud. Mao et al. [65] presented a resource-aware placement scheme to improve the system performance in a heterogeneous cluster. Nardelli et al. [66] studied the container placement problem for optimizing deployment cost. However, none of the above work considers the communication cost among containers.
- 32. 20 2 A Survey of Resource Management in Cloud and Edge Computing The container migration issues also have been extensively studied in the litera- ture. The first part of the related works concentrates on developing container live migration techniques. The works [67, 68] proposed solutions for live migrating Linux containers, while [69] proposed the techniques for live migrating Docker containers. The prior works [70–72] further optimized the existing container migration techniques for reducing migration overhead. The second part of the related works focused on the container migration strategies. Li et al. [73] aimed to achieve load balancing of cloud resources through container migration. Guo et al. [74] proposed a container scheduling strategy based on neighborhood division in micro service, with the purpose of reducing the system load imbalance and improve the overall system performance. Kaewkasi and Chuenmuneewong [75] applied the ant colony optimization (ACO) in the context of container scheduling, which aimed to balance the resource usages and achieve better performance. Xu et al. [76] proposed a resource scheduling approach for the container virtualized cloud environments to reduce response time of customers jobs and improve resource utilization. Again, none of the above work considers communication cost among containers. 2.4 Near-Optimal Network System for Data Replication Here we discuss some representative work that is related to BDS+ in three categories. Overlay Network Control Overlay networks realize great potential for various applications, especially for data transfer applications. The representative networks include peer-to-peer (P2P) networks and content delivery networks (CDNs). The P2P architecture has already been verified by many applications, such as live streaming systems (CoolStreaming [77], Joost [78], PPStream [79], UUSee [80]), video-on-demand (VoD) applications (OceanStore [81]), distributed hash tables [82], and more recently Bitcoin [83]. But, self-organizing systems based on P2P principles suffer from long convergence times. CDN distributes services spatially relative to end users to provide high availability and performance (e.g., to reduce page load time), serving many applications such as multimedia [84] and live streaming [85]. We briefly introduce the two baselines in the evaluation section: (1) Bullet [86], which enables geo-distributed nodes to self-organize into an overlay mesh. Specifically, each node uses RanSub [87] to distribute summary ticket information to other nodes and receive disjoint data from its sending peers. The main difference between BDS+ and Bullet lies in the control scheme, i.e., BDS+ is a centralized method that has a global view of data delivery states, while Bullet is a decentralized scheme and each node makes its decision locally. (2) Akamai designs a three- layer overlay network for delivering live streams [88], where a source forwards its streams to reflectors and reflectors send outgoing streams to stage sinks. There are two main differences between Akamai and BDS+. First, Akamai adopts a three-
- 33. 2.4 Near-Optimal Network System for Data Replication 21 layer topology where edge servers receive data from their parent reflectors, while BDS+ successfully explores a larger search space through a finer-grained allocation without the limitation of three coarse-grained layers. Second, the receiving sequence of data must be sequential in Akamai because it is designed for a live streaming application. But there is no such requirements in BDS+, and the side effect is that BDS+ has to decide the optimal transmission order as additional work. Data Transfer and Rate Control Rate control of transport protocols at the DC level plays an important role in data transmission. DCTCP [89], PDQ [90], CONGA [91], DCQCN [92], and TIMELY [93] are all classical protocols showing clear improvements in transmission efficiency. Some congestion control protocols like the credit-based ExpressPass [94] and load balancing protocols like Hermes [95] could further reduce flow completion time by improving rate control. On this basis, the recently proposed NUMfabric [96] and Domino [97] further explore the potential of centralized TCP on speeding up data transfer and improving DC throughput. To some extend, co-flow scheduling [98, 99] has some similarities to the multicast overlay scheduling, in terms of data parallelism. But that work focuses on flow-level problems, while BDS+ is designed at the application level. Centralized Traffic Engineering Traffic engineering (TE) has long been a hot research topic, and many existing studies [100–106] have illustrated the challenges of scalability, heterogeneity etc., especially on inter-DC level. The representative TE systems include Google’s B4 [107] and Microsoft’s SWAN [108]. B4 adopts SDN [109] and OpenFlow [110, 111] to manage individual switches and deploy customized strategies on the paths. SWAN is another online traffic engineering platform, which achieves high network utilization with its software-driven WAN. Network Change Detection Detecting network changes is quite important not only in traffic prediction problems, but also in many other applications, such as abnormality detection, network monitoring, and security. There are two basic but mature methods that are widely used, the exponentially weighted moving average (EWMA) control scheme [112, 113] and the change point detection algorithm [114]. EWMA usually gives higher weights to recent observations while gives decreased weights in geometric progression to the previous observations, when predicting the next value. Although EWMA describes a graphical procedure for generating geometric moving averages smoothly, it faces an essential sensitivity problem; in other words, it cannot identify abrupt changes. In contrast, change point detection algorithms could exactly solve this problem, in both online [115–117] and offline [118–121] manner. BDS+ combines these two methods by designing a sliding observation window, which makes BDS+’s prediction algorithm both stable and sensitive. Overall, an application-level multicast overlay network with dynamic bandwidth separation is essential for data transfer in inter-DC WANs. Applications like user logs, search engine indexes, and databases would greatly benefit from bulk-data multicast. Furthermore, such benefits are orthogonal to prior WAN optimizations, further improving inter-DC application performance.
- 34. 22 2 A Survey of Resource Management in Cloud and Edge Computing 2.5 Distributed Edge Caching in Short Video Network Here we discuss some representative caching policies in traditional CDN and some related edge caching systems. Existing caching policies The most representative caching policies such as FIFO, LRU, LFU, and their variations are simple but effective in traditional CDN, where the frequency of content visits can be modeled as Poisson distribution. Under these policies, the future popularity of a content is represented by the historical popularity. But in short video network, user access pattern is non-stationary and is no longer in Poisson distribution, so these policies become inefficiency in short video network. The same problem exists in the TTL (Time-to-live)-based caching policies. For example, in the caching (feedforward) networks, where the access pattern is Markov arrival process, [122] gives joint consideration about both TTL and request models and drives evictions by stopping times. Ferragut et al. [123] set an optimal timer to maximize cache hit rate, but it only works in the case of Pareto-distributed access pattern and Zipf distribution of file popularity and thus certainly becomes invalid in short video network. Basu et al. [124] propose two caches named f-TTL with two timer, so as to filter out non-stationary traffic, but it still relies on locally observed access patterns to change TTL values, regardless of the future popularity. One of the promising attempts in recent years is learning-based proactive prediction-based policy. Narayanan et al. [125] train a characteristics predictor to predict object future popularity and interoperate with traditional LRU and LFU, boosting the number of cache hits. However, it looks a fixed length into the future to predict object popularity (1–3, 12–14, and 24–26 h), ignoring the temporal and spatial video popularity pattern, thus cannot handle the varies life spans in short video network. Pensieve [126] trains a neural network model as an adaptive bitrate (ABR) algorithm; it complements our AutoSight framework, in the sense that the proposed dynamic control rules can give us a reference when facing the various life span problem, but it ignores temporal pattern information and only works in live streaming. Besides, [127] introduces reinforcement learning for making cache decisions, but it works in the case that user requests comply with the Markov process, while the proposes AutoSight in this book can work under arbitrary non- stationary user access patterns. Edge caching systems Edge computing was proposed to enable offloading of latency-sensitive tasks to edge servers instead of cloud and has achieved rapid development in many areas such as 5G, wireless, mobile networks [128], and video streaming [129]. Cachier [130] uses a caching model on edge servers to balance load between edge and cloud for image-recognition applications, but it doesn’t predict image loads in the future. Gabry et al. [131] study the content placement problem in edge caching to maximize energy efficiency; the analysis in this work gives us references to design our AutoSight network topology, but what we considered under such topology is the caching of short video network rather than energy-saving.
- 35. 2.6 The Controllability of Dynamic Temporal Network 23 Ma et al. [129] propose a geo-collaborative caching strategy for mobile video network, suggesting that joint caching over multiple edges can improve QoE, which provides strong proof for our AutoSight design. While this chapter tries to reveal the characteristics of different mobile videos, we focus the short video network on edge servers with unique user access pattern and video popularity pattern. 2.6 The Controllability of Dynamic Temporal Network Dynamic network The network is composed of nodes and the relationship between nodes. The rapid development of the Internet makes human beings enter the network era, such as all kinds of social networks. The industrial Internet such as large power grid and the Internet of things, connects the whole world into a huge network. At present, network research has infiltrated into mathematics, life science, information science, and many other fields, and its fundamental goal is to find effective means to control network behavior and make it serve human beings. The paper [132] firstly applies the classical control theory to the analysis of network control. In a directed network, it uses a directed edge from node N1 to node N2 to denote that N2 can be controlled by N1 under some conditions. In this book, the network is abstracted into a linear time-invariant system, and the Kalman controllability criterion in the control theory is introduced into the research, and the sufficient and necessary condition for the network to achieve complete controllability is that the controllability matrix reaches full rank, so the controllability problem of the network is transformed into the problem of calculating the rank of the controllability matrix. What is mentioned in [133] is theoretically proved that the minimum driver nodes needed to control the entire network depend on the maximum matching in the network, in which all unmatching nodes are driver nodes of the network. Thus, the problem of network structure controllability is transformed into a classical graph theory matching problem, which reduces the time complexity of network controllability problem. However, this method can only be applied to directed networks and networks with unknown edge weights. According to Popov-Belevitch- Hautus criterion, the paper [134] has proved that the minimum number of driver nodes required by the network is equal to the maximum geometric multiplicity of all eigenvalues of the network matrix; as for undirected networks, the minimum driver nodes are the maximum algebraic multiplicity of all eigenvalues. It reduces the computational complexity of related problems greatly. References [135–138] presented a theoretical approach to describing the control- lability of networks or proposed a way to change the state of some nodes to stabilize the network. All of the papers and methods mentioned above have not solved the problem of controllability generated in dynamic networks well until now. Internet of Vehicles Internet of Vehicles refers to the use of vehicle electronic sensing devices to enable two-way data exchange and sharing between vehicles,
- 36. 24 2 A Survey of Resource Management in Cloud and Edge Computing Fig. 2.2 Illustration of vehicle-to-vehicle communication people, and transportation facilities through wireless communication technology, car navigation systems, intelligent terminal facilities, and information processing systems. It is comprehensive intelligent decision-making information system that realizes real-time monitoring, scientific dispatching, and effective management of vehicles, people, objects, roads, etc., thereby improving road transportation conditions and traffic management efficiency [139]. Vehicle-to-vehicle communication is shown in Fig. 2.2, the communication between the vehicle and the vehicle through the vehicle-mounted terminal, and it is mainly used for two-way data transmission between vehicles. The communication technologies used include microwave, infrared technology, dedicated short-range communication, etc., featuring high safety and real-time requirements. The vehicle terminals can collect information such as the speed, position, direction, and vehicle running alarm of the surrounding vehicles in real time. Those vehicles form an interactive communication platform through wireless communication technology, which can exchange pictures, text messages, videos, and audio information in real time [140, 141] (Fig. 2.3). The communication between the vehicle and the control center means that the vehicle-mounted mobile terminal establishes interconnection with the remote traffic control center through the public access network, to complete data transmission and information exchange and to accomplish the interaction and storage of data between the vehicle and the traffic control center. It is mainly used in vehicle navigation, vehicle remote monitoring, emergency rescue, information entertainment services, and so on. Moreover, it has the characteristics of long distance and high-speed movement [139]. In a vehicular network, vehicles are virtualized as mobile network nodes, and road side units (RSUs) are virtualized as stationary network nodes. The environmen- tal information of the road and the vehicle is collected through sensors in the vehicle and the RSU. The structure of the vehicle network presents a dynamic topology. The high-speed moving vehicle nodes make the topology of the vehicle network change
- 37. References 25 Fig. 2.3 Illustration of communication between the vehicle and the control center rapidly, and the access status changes dynamically due to the dynamic network topology. References 1. Mauve, M., Vogel, J., Hilt, V., Effelsberg, W.: Local-lag and timewarp: providing consistency for replicated continuous applications. IEEE Trans. Multimedia 6(1), 47–57 (2004) 2. Shao, Z., Jin, X., Jiang, W., Chen, M., Chiang, M.: Intra-data-center traffic engineering with ensemble routing. In: INFOCOM, 2013 Proceedings IEEE, pp. 2148–2156. IEEE (2013) 3. Webb, S.D., Soh, S., Lau, W.: Enhanced mirrored servers for network games. In: Proceedings of the 6th ACM SIGCOMM Workshop on Network and System Support for Games, pp. 117– 122. ACM (2007) 4. Vik, K.-H., Halvorsen, P., Griwodz, C.: Multicast tree diameter for dynamic distributed inter- active applications. In: INFOCOM 2008. The 27th Conference on Computer Communications IEEE. IEEE (2008) 5. Guo, J., Liu, F., Zeng, D., Lui, J.C., Jin, H.: A cooperative game based allocation for sharing data center networks. In: INFOCOM, 2013 Proceedings IEEE, pp. 2139–2147. IEEE (2013) 6. Xu, K., Zhang, Y., Shi, X., Wang, H., Wang, Y., Shen, M.: Online combinatorial double auc- tion for mobile cloud computing markets. In: Performance Computing and Communications Conference (IPCCC), 2014 IEEE International, pp. 1–8. IEEE (2014) 7. Seung, Y., Lam, T., Li, L.E., Woo, T.: Cloudflex: seamless scaling of enterprise applications into the cloud. In: INFOCOM, 2011 Proceedings IEEE, pp. 211–215. IEEE (2011) 8. Yue, K., Wang, X.-L., Zhou, A.-Y., et al.: Underlying techniques for web services: a survey. J. Softw. 15(3), 428–442 (2004)
- 38. 26 2 A Survey of Resource Management in Cloud and Edge Computing 9. Zaki, Y., Chen, J., Potsch, T., Ahmad, T., Subramanian, L.: Dissecting web latency in ghana. In: Proceedings of the 2014 Conference on Internet Measurement Conference, pp. 241–248. ACM (2014) 10. Pujol, E., Richter, P., Chandrasekaran, B., Smaragdakis, G., Feldmann, A., Maggs, B.M., Ng, K.-C.: Back-office web traffic on the Internet. In: Proceedings of the 2014 Conference on Internet Measurement Conference, pp. 257–270. ACM (2014) 11. Wang, H., Shea, R., Ma, X., Wang, F., Liu, J.: On design and performance of cloud-based distributed interactive applications. In: 2014 IEEE 22nd International Conference on Network Protocols (ICNP), pp. 37–46. IEEE (2014) 12. Dogar, F.R., Karagiannis, T., Ballani, H., Rowstron, A.: Decentralized task-aware scheduling for data center networks. ACM SIGCOMM Comput. Commun. Rev. 44, 431–442 (2014) 13. Alizadeh, M., Greenberg, A., Maltz, D.A., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., Sridharan, M.: Data center TCP (DCTCP). ACM SIGCOMM Comput. Commun. Rev. 41(4), 63–74 (2011) 14. Vamanan, B., Hasan, J., Vijaykumar, T.: Deadline-aware datacenter TCP (D2TCP). ACM SIGCOMM Comput. Commun. Rev. 42(4), 115–126 (2012) 15. Wilson, C., Ballani, H., Karagiannis, T., Rowtron, A.: Better never than late: meeting deadlines in datacenter networks. ACM SIGCOMM Comput. Commun. Rev. 41(4), 50–61 (2011). ACM 16. Hong, C.-Y., Caesar, M., Godfrey, P.: Finishing flows quickly with preemptive scheduling. ACM SIGCOMM Comput. Commun. Rev. 42(4), 127–138 (2012) 17. Dukkipati, N., McKeown, N.: Why flow-completion time is the right metric for congestion control. ACM SIGCOMM Comput. Commun. Rev. 36(1), 59–62 (2006) 18. Alizadeh, M., Yang, S., Sharif, M., Katti, S., McKeown, N., Prabhakar, B., Shenker, S.: pFabric: minimal near-optimal datacenter transport. ACM SIGCOMM Comput. Commun. Rev. 43(4), 435–446 (2013). ACM 19. Zhang, H.: More load, more differentiation – a design principle for deadline-aware flow control in DCNS. In: INFOCOM, 2014 Proceedings IEEE. IEEE (2014) 20. Shen, M., Gao, L., Xu, K., Zhu, L.: Achieving bandwidth guarantees in multi-tenant cloud networks using a dual-hose model. In: 2014 IEEE 33rd International Performance Computing and Communications Conference (IPCCC), pp. 1–8. IEEE (2014) 21. Xu, K., Zhang, Y., Shi, X., Wang, H., Wang, Y., Shen, M.: Online combinatorial double auction for mobile cloud computing markets. In: 2014 IEEE 33rd International Performance Computing and Communications Conference (IPCCC), pp.1–8. IEEE (2014) 22. Gavish, B., Pirkul, H.: Computer and database location in distributed computer systems. IEEE Trans. Comput. (7), 583–590 (1986) 23. Gavish, B., Pirkul, H.: Algorithms for the multi-resource generalized assignment problem. Manag. Sci. 37(6), 695–713 (1991) 24. Ross, G.T., Soland, R.M.: A branch and bound algorithm for the generalized assignment problem. Math. Program. 8(1), 91–103 (1975) 25. Oncan, T.: A survey of the generalized assignment problem and its applications. INFOR 45(3), 123–141 (2007) 26. Privault, C., Herault, L.: Solving a real world assignment problem with a metaheuristic. J. Heuristics 4(4), 383–398 (1998) 27. Mitrović-Minić, S., Punnen, A.P.: Local search intensified: very large-scale variable neigh- borhood search for the multi-resource generalized assignment problem. Discret. Optim. 6(4), 370–377 (2009) 28. Yagiura, M., Iwasaki, S., Ibaraki, T., Glover, F.: A very large-scale neighborhood search algorithm for the multi-resource generalized assignment problem. Discret. Optim. 1(1), 87–98 (2004) 29. Mazzola, J.B., Wilcox, S.P.: Heuristics for the multi-resource generalized assignment prob- lem. Nav. Res. Logist. 48(6), 468–483 (2001) 30. Shtub, A., Kogan, K.: Capacity planning by the dynamic multi-resource generalized assign- ment problem (DMRGAP). Eur. J. Oper. Res. 105(1), 91–99 (1998)
- 39. References 27 31. Gavranović, H., Buljubašić, M.: An efficient local search with noising strategy for Google machine reassignment problem. Ann. Oper. Res. 242, 1–13 (2014) 32. Sharma, P., Chaufournier, L., Shenoy, P., Tay, Y.C.: Containers and virtual machines at scale: a comparative study. In: International Middleware Conference, p. 1 (2016) 33. Mann, Z.D., Szabó, M.: Which is the best algorithm for virtual machine placement optimiza- tion? Concurr. Comput. Pract. Exp. 29(7), e4083 (2017) 34. Meng, X., Pappas, V., Zhang, L.: Improving the scalability of data center networks with traffic-aware virtual machine placement. In: INFOCOM, 2010 Proceedings IEEE, pp. 1–9 (2010) 35. Popa, L., Kumar, G., Chowdhury, M., Krishnamurthy, A., Ratnasamy, S., Stoica, I.: Faircloud: sharing the network in cloud computing. ACM SIGCOMM Comput. Commun. Rev. 42(4), 187–198 (2012) 36. Lacurts, K., Deng, S., Goyal, A., Balakrishnan, H.: Choreo: network-aware task placement for cloud applications. In: Conference on Internet Measurement Conference, pp. 191–204 (2013) 37. Li, X., Wu, J., Tang, S., Lu, S.: Let’s stay together: towards traffic aware virtual machine placement in data centers. In: INFOCOM, 2014 Proceedings IEEE, pp. 1842–1850 (2014) 38. Ma, T., Wu, J., Hu, Y., Huang, W.: Optimal VM placement for traffic scalability using Markov chain in cloud data centre networks. Electron. Lett. 53(9), 602–604 (2017) 39. Zhao, Y., Huang, Y., Chen, K., Yu, M., Wang, S., Li, D.S.: Joint VM placement and topology optimization for traffic scalability in dynamic datacenter networks. Comput. Netw. 80, 109– 123 (2015) 40. Rai, A., Bhagwan, R., Guha, S.: Generalized resource allocation for the cloud. In: ACM Symposium on Cloud Computing, pp. 1–12 (2012) 41. Wang, L., Zhang, F., Aroca, J.A., Vasilakos, A.V., Zheng, K., Hou, C., Li, D., Liu, Z.: Greendcn: a general framework for achieving energy efficiency in data center networks. IEEE J. Sel. Areas Commun. 32(1), 4–15 (2013) 42. Rui, L., Zheng, Q., Li, X., Jie, W.: A novel multi-objective optimization scheme for rebal- ancing virtual machine placement. In: IEEE International Conference on Cloud Computing, pp. 710–717 (2017) 43. Gu, L., Zeng, D., Guo, S., Xiang, Y., Hu, J.: A general communication cost optimization framework for big data stream processing in geo-distributed data centers. IEEE Trans. Comput. 65(1), 19–29 (2015) 44. Shen, M., Xu, K., Li, F., Yang, K., Zhu, L., Guan, L.: Elastic and efficient virtual network provisioning for cloud-based multi-tier applications. In: 2015 44th International Conference on Parallel Processing (ICPP), pp. 929–938. IEEE (2015) 45. Wang, T., Xu, H., Liu, F.: Multi-resource load balancing for virtual network functions. In: IEEE International Conference on Distributed Computing Systems (2017) 46. Taleb, T., Bagaa, M., Ksentini, A.: User mobility-aware virtual network function placement for virtual 5G network infrastructure. In: IEEE International Conference on Communications, pp. 3879–3884 (2016) 47. Mehraghdam, S., Keller, M., Karl, H.: Specifying and placing chains of virtual network functions. In: 2014 IEEE 3rd International Conference on Cloud Networking (CloudNet), pp. 7–13. IEEE (2014) 48. Kawashima, K., Otoshi, T., Ohsita, Y., Murata, M.: Dynamic placement of virtual network functions based on model predictive control. In: NOMS 2016 – 2016 IEEE/IFIP Network Operations and Management Symposium, pp. 1037–1042 (2016) 49. Marotta, A., Kassler, A.: A power efficient and robust virtual network functions placement problem. In: Teletraffic Congress, pp. 331–339 (2017) 50. Addis, B., Belabed, D., Bouet, M., Secci, S.: Virtual network functions placement and routing optimization. In: IEEE International Conference on Cloud NETWORKING, pp. 171–177 (2015)
- 40. 28 2 A Survey of Resource Management in Cloud and Edge Computing 51. Wang, F., Ling, R., Zhu, J., Li, D.: Bandwidth guaranteed virtual network function placement and scaling in datacenter networks. In: IEEE International Performance Computing and Communications Conference, pp. 1–8 (2015) 52. Ghaznavi, M., Khan, A., Shahriar, N., Alsubhi, K., Ahmed, R., Boutaba, R.: Elastic virtual network function placement. In: IEEE International Conference on Cloud Networking (2015) 53. Bhamare, D., Samaka, M., Erbad, A., Jain, R., Gupta, L., Chan, H.A.: Optimal virtual network function placement in multi-cloud service function chaining architecture. Comput. Commun. 102(C), 1–16 (2017) 54. Zhang, Q., Xiao, Y., Liu, F., Lui, J.C.S., Guo, J., Wang, T.: Joint optimization of chain placement and request scheduling for network function virtualization. In: IEEE International Conference on Distributed Computing Systems, pp. 731–741 (2017) 55. Taleb, T., Bagaa, M., Ksentini, A.: User mobility-aware virtual network function placement for virtual 5G network infrastructure. In: 2015 IEEE International Conference on Communi- cations (ICC), pp. 3879–3884. IEEE (2015) 56. Laghrissi, A., Taleb, T., Bagaa, M., Flinck, H.: Towards edge slicing: VNF placement algorithms for a dynamic & realistic edge cloud environment. In: 2017 IEEE Global Communications Conference, pp. 1–6. IEEE (2017) 57. Prados, M.B.J., Laghrissi, A., Taleb, A.T., Taleb, T., Bagaa, M., Flinck, H.: A queuing based dynamic auto scaling algorithm for the LTE EPC control plane. In: 2018 IEEE Global Communications Conference, pp. 1–6. IEEE (2018) 58. Bagaa, M., Taleb, T., Ksentini, A.: Service-aware network function placement for efficient traffic handling in carrier cloud. In: 2014 IEEE Wireless Communications and Networking Conference (WCNC), pp. 2402–2407. IEEE (2014) 59. Bagaa, M., Dutra, D.L.C., Addad, R.A., Taleb, T., Flinck, H.: Towards modeling cross-domain network slices for 5G. In: 2018 IEEE Global Communications Conference, pp. 1–6. IEEE (2018) 60. Bagaa, M., Taleb, T., Laghrissi, A., Ksentini, A.: Efficient virtual evolved packet core deployment across multiple cloud domains. In: 2018 IEEE Wireless Communications and Networking Conference (WCNC), pp. 1–6. IEEE (2018) 61. Zhang, R., Zhong, A.-M., Dong, B., Tian, F., Li, R.: Container-VM-PM architecture: a novel architecture for docker container placement. In: International Conference on Cloud Computing, pp. 128–140. Springer (2018) 62. Piraghaj, S.F., Dastjerdi, A.V., Calheiros, R.N., Buyya, R.: A framework and algorithm for energy efficient container consolidation in cloud data centers. In: 2015 IEEE International Conference on Data Science and Data Intensive Systems (DSDIS), pp. 368–375. IEEE (2015) 63. Dong, Z., Zhuang, W., Rojas-Cessa, R.: Energy-aware scheduling schemes for cloud data centers on google trace data. In: 2014 IEEE Online Conference on Green Communications (OnlineGreencomm), pp. 1–6. IEEE (2014) 64. Shi, T., Ma, H., Chen, G.: Energy-aware container consolidation based on PSO in cloud data centers. In: 2018 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. IEEE (2018) 65. Mao, Y., Oak, J., Pompili, A., Beer, D., Han, T., Hu, P.: Draps: dynamic and resource-aware placement scheme for docker containers in a heterogeneous cluster. In: 2017 IEEE 36th International Performance Computing and Communications Conference (IPCCC), pp. 1–8. IEEE (2017) 66. Nardelli, M., Hochreiner, C., Schulte, S.: Elastic provisioning of virtual machines for container deployment. In: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion, pp. 5–10. ACM (2017) 67. Qiu, Y.: Evaluating and improving LXC container migration between cloudlets using multipath TCP. Ph.D. dissertation, Carleton University, Ottawa (2016) 68. Machen, A., Wang, S., Leung, K.K., Ko, B.J., Salonidis, T.: Live service migration in mobile edge clouds. IEEE Wirel. Commun. 25(1), 140–147 (2018) 69. Pickartz, S., Eiling, N., Lankes, S., Razik, L., Monti, A.: Migrating Linux containers using CRIU. In: International Conference on High Performance Computing, pp. 674–684. Springer (2016)
- 41. References 29 70. Ma, L., Yi, S., Carter, N., Li, Q.: Efficient live migration of edge services leveraging container layered storage. IEEE Trans. Mob. Comput. 18, 2020–2033 (2018) 71. Ma, L., Yi, S., Li, Q.: Efficient service handoff across edge servers via docker container migration. In: Proceedings of the Second ACM/IEEE Symposium on Edge Computing, p. 11. ACM (2017) 72. Nadgowda, S., Suneja, S., Bila, N., Isci, C.: Voyager: complete container state migration. In: 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pp. 2137–2142. IEEE (2017) 73. Li, P., Nie, H., Xu, H., Dong, L.: A minimum-aware container live migration algorithm in the cloud environment. Int. J. Bus. Data Commun. Netw. (IJBDCN) 13(2), 15–27 (2017) 74. Guo, Y., Yao, W.: A container scheduling strategy based on neighborhood division in micro service. In: NOMS 2018-2018 IEEE/IFIP Network Operations and Management Symposium, pp. 1–6. IEEE (2018) 75. Kaewkasi, C., Chuenmuneewong, K.: Improvement of container scheduling for docker using ant colony optimization. In: 2017 9th International Conference on Knowledge and Smart Technology (KST), pp. 254–259. IEEE (2017) 76. Xu, X., Yu, H., Pei, X.: A novel resource scheduling approach in container based clouds. In: 2014 IEEE 17th International Conference on Computational Science and Engineering (CSE), pp. 257–264. IEEE (2014) 77. Zhang, X., Liu, J., Li, B., Yum, Y.-S.: CoolStreaming/DONet: a data-driven overlay network for peer-to-peer live media streaming. In: INFOCOM, vol. 3, pp. 2102–2111. IEEE (2005) 78. Joost: http://www.joost.com/ 79. Ppstream: http://www.ppstream.com/ 80. Uusee: http://www.uusee.com/ 81. Oceanstore: http://oceanstore.cs.berkeley.edu/ 82. Rhea, S., Godfrey, B., Karp, B., Kubiatowicz, J., Ratnasamy, S., Shenker, S., Stoica, I., Yu, H.: Opendht: a public DHT service and its uses. In: ACM SIGCOMM, vol. 35, pp. 73–84 (2005) 83. Eyal, I., Gencer, A.E., Sirer, E.G., Van Renesse, R.: Bitcoin-NG: a scalable blockchain protocol. In: NSDI (2016) 84. Zhu, W., Luo, C., Wang, J., Li, S.: Multimedia cloud computing. IEEE Signal Process. Mag. 28(3), 59–69 (2011) 85. Sripanidkulchai, K., Maggs, B., Zhang, H.: An analysis of live streaming workloads on the Internet. In: IMC, pp. 41–54. ACM (2004) 86. Kostić, D., Rodriguez, A., Albrecht, J., Vahdat, A.: Bullet: high bandwidth data dissemination using an overlay mesh. ACM SOSP 37(5), 282–297 (2003). ACM 87. Rodriguez, A., Albrecht, J., Bhirud, A., Vahdat, A.: Using random subsets to build scalable network services. In: USITS, pp. 19–19 (2003) 88. Andreev, K., Maggs, B.M., Meyerson, A., Sitaraman, R.K.: Designing overlay multicast networks for streaming. In: SPAA, pp. 149–158 (2013) 89. Alizadeh, M., Greenberg, A., Maltz, D.A., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., Sridharan, M.: Data center TCP (DCTCP). In: ACM SIGCOMM, pp. 63–74 (2010) 90. Hong, C.Y., Caesar, M., Godfrey, P.B.: Finishing flows quickly with preemptive scheduling. ACM SIGCOMM Comput. Commun. Rev. 42(4), 127–138 (2012) 91. Alizadeh, M., Edsall, T., Dharmapurikar, S., Vaidyanathan, R., Chu, K., Fingerhut, A., Lam, V.T., Matus, F., Pan, R., Yadav, N.: CONGA: distributed congestion-aware load balancing for datacenters. In: ACM SIGCOMM, pp. 503–514 (2014) 92. Zhu, Y., Eran, H., Firestone, D., Guo, C., Lipshteyn, M., Liron, Y., Padhye, J., Raindel, S., Yahia, M.H., Zhang, M.: Congestion control for large-scale RDMA deployments. ACM SIGCOMM 45(5), 523–536 (2015) 93. Mittal, R., Lam, V.T., Dukkipati, N., Blem, E., Wassel, H., Ghobadi, M., Vahdat, A., Wang, Y., Wetherall, D., Zats, D.: TIMELY: RTT-based congestion control for the datacenter. In: ACM SIGCOMM, pp. 537–550 (2015)
- 42. Another Random Document on Scribd Without Any Related Topics
- 43. kunnes pienen sillan tuolla puolen poliisikonttori löytyi vanhanaikaisesta pienoisesta rakennuksesta. Jotenkin pian siellä saatiin kirjoitus passeihin, mutta puheesta ei tahtonut paljon tolkkua tulla, kun emme osanneet venäjää. Päällikkö oli upseeri everstin univormussa; kirjureita oli puoli tusinaa, ja niiden huono vaatteus oli silmäänpistävä. Mitään maksua ei kirjoituksesta otettu. Kun illempana toistamiseen lähdimme kaupungille, tuli kaduilla taas vastaamme muutamia oudonnäköisiä miehiä, jotka jo päivällä olimme huomanneet ja jotka näyttivät kuljeksivan toimettomina. Varsinkin kaksi heistä veti huomion puoleensa: toinen oli nuori, pitkä, komea, mustaverinen mies, päässä tataarilainen piippalakki ja nutun rinnus täynnä pikku kurttuja patruunain säilyttämistä varten, toinen vanha ukko, jolla oli korkea könkönenä, lumivalkoinen parta ja päässä lammasnahkalakki, sen näköinen ihan täsmälleen kuin turkkilaiset vanhukset kuvakirjoissa. Nämä muukalaiset olivat, niinkuin kestikievarissa tarkemmin saimme kuulla, niitä Kaukaasian tsherkessejä, jotka viime sodan aikana olivat nostaneet kapinan Venäjää vastaan ja rangaistukseksi nyt olivat siirretyt tänne äärimmäiseen pohjolaan; Kemissä heitä oli 20 miestä. Kruunulta he saivat muutamia kopeekoita päivässä, muuten elivät kerjuulla; isäntämme valitti heitä laiskoiksi. Vilulle kuuluivat olevan hyvin arat. Majoitettuina sinne tänne taloihin heillä oli täysi vapaus liikkua kaupungilla, kun ei tarvinnut pelätä, että he varojen puutteessa voisivat ajatella mitään pakoa. Kemissä muuten oli sotaväkeä 60 miestä. Matkallamme kävimme kaupungin rannassa katselemassa uutta kirkkoa, jota hyvin auliisti näyteltiin ja joka oli jokseenkin korea; sitten yritimme muutamaan taloon, jossa sanottiin kaupunkilaisten klubin olevan, mutta se oli kiinni. Viimein kuljettuamme kaupungin
- 44. läpi ristiin rastiin minä kiipesin oppaamme, erään kestikievarissa palvelevan jyskyjärveläisen Huotarin kanssa kaupungin etelärannasta kohoaville korkeille kallioille näköalaa laajemmalti katselemaan ja luodakseni silmäyksen itään päin Vienanmerta kohti, jonka laineita niin monta vuotta olin hartaasti halunnut nähdä. Aavempi meri tulee kuitenkin, niinkuin jo on mainittu, penikulman päähän Kemistä, niin etten muista tarkkaan, erotimmeko isosti varsinaista meren selkää. Kumppalini B. ei huolinut muka turhaan vaivata itseään tälle matkalle, vaan palasi takaisin kestikievariin, johon mekin sitten kulkumme käänsimme. Illalla majatalossa sovittiin isännän kanssa venekyydistä Solovetsin luostariin seuraavaksi päiväksi. 7 1/2 ruplasta hän lupasi meille laittaa veneen ja kuusi soutumiestä paitsi peränpitäjää, jotka veisivät meidät luostariin, viipyisivät siellä pari kolme päivää ja sitten toisivat meidät takaisin. Kahdentoista penikulman matkasta (yhteensä edestakaisin) tuo summa, kurssin mukaan noin 18 markkaa, ei tuntunut erittäin suurelta, vaan pikemmin halvalta, ja kun siihen tingimme isäntämme ensi vaatimuksen, 8 ruplaa, se tapahtui vain ilman aikojaan lystin vuoksi; että omatunto olisi rauhallisempi.
- 45. IV. Pyhä kaupunki. Matka sinne. "Pyhäksi kaupungiksi" ja "monasteriksi" nimittävät karjalaiset Solovetsin luostaria Vienanmeressä. Jälkimmäinen nimitys on ehkä tavallisempi, kuitenkin edellistä omituisuutensa vuoksi käytän tässä luvun päällekirjoituksena. Luostari on kuuluisa paikka, näkemältä tunnettu suurelle osalle Venäjän kansaa ja nimeltänsä myöskin sivistyneelle ulkomaalle. Karjalalle se yhä vielä on hengellisen elämän keskus, niinkuin aikoinaan oli maallisenkin, ja joka Karjalassa vaeltaa ilman tarkkaa matkan määrää, sen askeleet aivan itsestänsä kääntyvät luostaria kohti. "Tottahan monasteria käytte katsomassa", oli tavallinen kysymys tai arvelu, kun matkamme tarkoitusta tiedusteltiin, ja tähän tiedusteluun lisättiin aina joku suurinta ihailua osoittava ylistyslause. Karjalan köyhää maata ei kannattanut tulla katsomaan, mutta monasteria, se oli toinen asia, sen näkeminen kyllä maksoi vaivan. Kenties joku epäluulon kauna alkoi mielessämme liikkua, kun näitä ylenmääräisiä kiitosvirsiä alinomaa
- 46. saimme kuulla, mutta uteliaisuutemme oli kuitenkin hyvässä vireessä, kun 10 p. heinäk. laittausimme taipaleelle sinne. Vienanmerellä on joitakuita vuosia ollut säännöllinen höyrylaivaliikenne, jonka eräs "Arkangelin-Muurmannin" höyrylaivayhtiö Vienassa on pannut toimeen. Sitä ylläpiti tänä kesänä kaksi yhtiön höyryä "Onega" ja "Kemi", joista edellinen kulki pitkin Karjalan rannikkoa Kannanlahdesta aina Onegan kaupunkiin saakka, jälkimmäinen Kemistä Vienaan; yhtymäpaikka oli Kemin reti. Sen puolesta tämä kulku vielä kuitenkin oli vajanainen ja vähemmin tyydyttävä, että laivat ainoastaan joka kymmenes päivä kohtasivat toisensa, ja kun emme sattuneet tulemaan semmoiseen aikaan, täytyi meidän soutuveneellä yrittää luostarisaareen.[11] Ennenkuin päästiin lähtemään, alkoi kello jo lähetä 11 ap. Me olimme kyllä aikoja olleet valmiit ja aloimme jo pitkäksyä odotusta, mutta talonväen puolelta kesti yhä ahkeraa hommaa. Syyksi viipymiseen sanottiin, että soutajat eivät olleet valmiit, mutta todellinen syy lienee ollut, että talon kaksi lasta, poika ja tyttö, myöskin laitettiin matkaan, vaikka siitä vasta rannassa kuulimme, ja että isäntäväkeämme vaivasi saamattomuus, josta paluumatkalla saimme loistavia todistuksia. Viimein astuttiin alas rantaan yläpuolelle saarisiltaa, missä vene oli. Meitä karttui tässä sievä joukko matkalle-lähtijöitä: kuusi vaimonpuolista soutajaa, perämiehenä Huotari, Miikkula, jonka minä olin halunnut saada oppaaksi mukaan, hänen turvissaan hänen kaksi näppärää, vaikka vähän vallatonta veljenlastaan, me molemmat ja sitten eräs köyhä kerjäläispoika, jolle perämies omasta mahtipontisuudestaan oli antanut luvan tulla mukaan. Nämä 13 henkeä olivat näkyvissä, mutta yhden ainakin vielä tiesimme olevan veneessä, sillä isäntämme oli aamulla pyytänyt eräälle karjalaiselle tuttavalleen lupaa päästä
- 47. kulkemaan seurassamme, jota emme olleet kieltäneet. Vene eli "karpaso" oli vankkatekoinen, melkein kuin meidän laivanparkassit, ja sen peräpuolessa oli sylen levyinen katto; tämän katon alla arvasimme aivan oikein miehen olevan. Mutta emmepä voineet arvata, ennenkuin matkalla havaitsimme, että hänen vaimonsakin seurasi myötä; siis meitä yhteensä tuli 15 henkeä. Juuri kun venettä aiottiin lykätä rannasta, hoksasi perämies, että "komppaso", kompassi, oli unohtunut kotia, ja se oli siis vielä kiireen kautta noudettava, ennenkuin maista erkauttiin. Kulkuväylä Kemistä on sangen kaunis, ensin pitkin kapeahkoa merenlahtea ja sitten läpi saariston, jonka äärimmäiseen reunaan luetaan kaupungista kolme penikulmaa. Luonto kuitenkin on yksin pitänyt kauneudesta huolta; ihmiskäden avusta, esim. huviloista tai muusta semmoisesta, ei näy vähintäkään merkkiä. Jonkun neljänneksen päässä kaupungista näkyi v. 1854 rakennettujen patterien jätteitä muutamalla korkealla niemellä. Penikulman päässä kohotti pienoisella saarella jo mainittu höyrysaha piippunsa ilmaan, mutta siinä ei ollut mitään liikettä. Etelästä päin siinti useita korkeita kukkuloita (mannermaalla) ja saaria; edellisistä korkeimman nimen ilmoitti Huotari olevan karjalaksi Rievesän, venäjäksi Keljakan, vuori, johon Kemistä tulee 1 1/2 penik. Höyrysahan ulkopuolella alkoivat selät väljetä harvenevain saarten välissä, ja veden vihreä väri samoin kuin hyvin suolainen maku — en näet malttanut olla sitä maistamatta — vakuuttivat, että olimme nyt kyntämässä varsinaisen Vienanmeren laineita. Kumppalini B. oli toisten matkustajain esimerkin mukaan heti alusta matkaa vetäytynyt karpason kannen alle, mutta minä jäin sen päälle istuskelemaan, kaunista ilmaa ihannellen ja ympärilleni katsellen, onnellisena siitä ajatuksesta, että nyt vihdoin viimein sain kulkea Suomen itäistä rajamerta. Meidän oli määrä käydä maissa ja syödä päivällistä ulommaisimmalla saarella,
- 48. mutta kun sitä lähenimme, herätti toinen vähän sisempänä ja matkastamme etelään päin oleva saari huomioni; sen alastomalla kalliohuipulla oli näet paljon vierinkiviä, jotka etäältä saattoivat vilkkaalle mielikuvitukselle näyttää jos jonkin muotoisilta. Kysymykseeni vastasi Huotari, että saari oli keskimmäinen Gusovoin eli (Hanhi-) saari, ja selitti sitten suurella vakaumuksella, että luostarin vihollinen oli kerran tullut tälle saarelle ja huipulle nousten uhitellut luostaria, jotta "et enää kauan siellä kiillä", mutta seuraus oli ollut, että kaikki viholliset muuttuivat kiviksi ja vielä tänä päivänä kivettyneinä istuvat kalliolla. Tämmöistä ihmettä näkemään tietysti jokainen olisi ollut utelias, ja siksi heti pyysin, että noustaisiin maihin tähän saareen, johon veneväki suostuikin, vähän ensin esteltyänsä, sillä pakovesi kantoi meitä toisaanne päin. Saari — jota luostarikirjoissa nimitetään Kusovoin saareksi ja jonka nimen alkujuuri siis ehkä yhtä hyvin voipi olla kuusi kuin hanhi (ven. gus) — näytti vähän suuremmalta ja korkeammalta kuin Linnasaari Oulussa; sen itäpäässä oli vähäinen notko, jossa kasvoi matalia puita ja pensaita, muuten se oli alaston kallio. Kun notkon kohdalla oli laskettu rantaan, nousimme B. ja minä ensin notkon ylimmälle paikalle ja aloimme sitten kiivetä lännen puoleen tulevan kallion jyrkkää rinnettä ylöspäin. Silmäni olivat kokonaan pettäneet minut saaren korkeuden ja suuruuden suhteen, sillä jo tämä ensimmäinen rinne oli niin suunnattoman korkea, että sen päälle päästessä läähätimme aikalailla, ja kun ylös tultua toivoimme olevamme saaren ylimmällä paikalla, havaitsimmekin, että kalliotanner yhä, vaikka loivemmin, kohosi etelään päin, johon ulottui pitkän matkaa. Tällä rinteellä näkyi lukematon joukko pienempiä ja suurempia soikulaisia kiviä, mutta jo ensi silmäys riitti vakuuttamaan, että noiden erityisten, ihmisenmuotoisten löytäminen ilman opasta tässä paljoudessa vaatisi puolen, jos ei koko päivää, niin ettemme
- 49. hakemista yrittäneetkään. Hiukka harmittavaista oli ensin, että turhaa vaivaa olimme nähneet, mutta kohta lohdutti meitä se ajatus, joka heti alusta oli varmana ollut mielessämme, että tuo taru luostarin vihollisista tällä saarella oli kaikkea todellista pohjaa vailla; sillä mitkä viholliset täältä olisivat luostaria ahdistaneet? Sen sijaan että olisimme tuota kivikenttää ruvenneet kulkemaan ristiin rastiin oudonnäköisiä kivimuodostuksia löytääksemme, käännyimme selin siihen ja aloimme katsella allamme ja edessämme itään ja pohjaan päin aukenevaa laajaa näköalaa. Jälkimmäiselle ilmansuunnalle tuli saariston vihannuus, edelliselle meren sinertävä, ääretön vedenpinta helottaen auringon kirkkaassa valossa. Koilliseen päin ei merellä ollut muuta rajaa kuin taivas, mutta äärimmäisenä idässä näkyi mustempi viiva ja siinä eräällä kohtaa valkea pilkku. Siellä oli matkamme määrä: tuo musta viiva oli monasterin saari ja tuo valkea pilkku itse luostari. Näköalan laajuus korvasi täydellisesti sen vähäisen ponnistuksen, minkä ylösnouseminen oli vaatinut: Vienanmeri, Vienanmeri, se ikävöity ja kauan kaivattu, tuossahan se nyt viimein levitteli ulapoitaan aukeoita, lakehia laineitansa ihastuneiden silmäimme edessä. Kun hetken takaa kuulimme huudettavan alas syömään, laskeusimme tyytyväisinä notkolle ja rantaan, surematta jättäen nuo kivettyneet ihmiset asemillensa. Jos kuitenkin silloin olisin tiennyt, mitä luostarissa ostamastani kirjasta jälkeenpäin tulin tietämään, — ja josta Castrénkin otteessaan luostarikronikasta ("Suomi", 1843) mainitsee, vaikk'en sitä jaksanut muistaa, — en varmaan niin vähällä olisi hakemistyöstä luopunut. Näillä Kusovoin saarilla oli tosiaan kerran majaillut luostarin vihollisia, ja ne viholliset olivat suomalaisia! V. 1611 tuli nimittäin suomalainen retkikunta rajan poikki, hävitti luostarille kuuluvat volostit (piirikunnat) Vienanmeren rannalla ja kulki sitten pienillä veneillä mainituille saarille saakka, jotka ovat noin kolmen
- 50. penikulman päässä luostarista. Täällä retkikunta viipyi kauan aikaa, mutta ei voinut luostaria saada valtaansa, "koska Jumalan näkymätön voima ja Solovetsin ihmeidentekijäin rukoukset varjelivat sitä ja sokaisivat viholliset", sanoo luostarikronikka, ja kun venäläinen sotaväki Suman linnasta riensi vastaan, täytyi retkikunnan palata ensin mannermaalle ja sitten kotia; Venäjän ja Ruotsin välillä oli nimittäin silloin tehty aselepo. Tähän historialliseen tapaukseen tuo Huotarin kertoma muinaistaru nähtävästi perustui. Leirinsä suomalaisilla oli arvatenkin ollut tällä samalla saarella, jossa nyt kävimme, ja sen huipulta he varmaan monta kertaa olivat heittäneet halukkaita silmäyksiä tuon leveän merenselän poikki luostarisaareen, jota kuitenkaan eivät likempää saaneet nähdä. Kun tuo taru näin koskee meitä itseämme, harmittaa minua nyt jälestäpäin, etten sen hartaammin kokenut saada selkoa noista kivettyneistä esi-isistämme, kun olin heitä niin lähellä, tuskin edes kivenheiton päässä. Jos joku tämän kertomuksen lukija joskus sattumalta sortuisi niille maille, kehoittaisin häntä käyttäytymään älykkäämmin. Syötyä lähdettiin heti matkaan. Emme kuitenkaan olleet pitkälle ennättäneet, ennenkuin meidät saavutti paksu sumu eli "turnano", niinkuin sanottiin, ja rankka sadekuuro. Jälkimmäinen pakotti meidät kaikki, jotka vain mahduimme, pakenemaan veneen kannen alle, jossa hätä yritti vallalle päästä, kun huomattiin, että ravistunut katto ei vettä pitänytkään; mutta onneksi sade yhtä äkkiä herkesi kuin oli tullutkin. Sumun tähden oli laskettu ulommaisen saaren rantaan ja siinä yhä istuttiin, vaikka aurinkokin rupesi taas paistamaan, niin että jo alkoi ikäväksi käydä, jonka vuoksi Huotarilta, joka "Norveegiasta" (Norjasta) ostettu "sydvesti" päässä ja öljytakki yllään totisena kökötti peräsimessä, viimein kyllästyksissäni kysyin, mitä tässä
- 51. viivyttiin ja miksi ei lähdetty matkaan. Huotari silloin selitti, että Kemistä mukaamme tullut uusi karjalainen matkakumppalimme, jonka käskystä saareen oli menty, oli noussut saaren huipulle tähystelemään näköalaa. "Mikähän mies se on", arvelin närkästyneenä B:lle, "joka hyvyydestä otetaan mukaan ja käyttäytyy kuin isäntä ja komentaja"; sillä muistin samalla, kuinka B. alusta matkaa oli kummastellut, että sanottu mies vaimoineen oli kannen alla ollut kuin kotonaan ja herrana, vaikka tosin kyllä oli Bilekin sijaa vieressään antanut. Viimein hän näkyi kalliosaaren korkealla huipulla, josta kiireesti alkoi astua alas. Kun hän kalliolta hypähti veneeseen ja käski lähteä liikkeelle, en voinut olla kysymättä, missä hän oli ollut, kun niin kauan oli kulkua viivyttänyt, johon hän vähän terävästi vastasi käyneensä katsomassa, kuinka paksu sumu merellä päin oli ja näkyikö luostarin saari. "Onhan meillä komppaso muassa", väitin minä tähän, muistuttaen että tämä tarvekalu juuri lähtiessä oli noudettu mukaan. "Semmoinen komppaso", vastasi mies ylenkatseellisesti, "sillä ei tumanossa tee mitään"; ja Huotari, joka näytti olevan valmis täydellisesti asettumaan miehen komennon alle, myönsi, että rauta-aineet veneessä kokonaan häiritsivät komppasomme toimintaa. Tähän ei ollut mitään sanomista, ja kun mies oli selvittänyt, että kolmen penikulman selälle ei ollut viisasta lähteä sumussa ilman täyttä ilmansuunnan tietoa, sain olla niinkuin aloinkin olla tyytyväinen. Hän sitten määräsi suunnan, johon oli kuljettava, ja niin jonkun tunnin viivähdyksen jälkeen taas oltiin matkassa. Tämä mies, jonka kanssa ensimmäinen kohtaukseni ei ollut aivan ystävällistä laatua, oli laivankippari nimeltä Nikolai Andrejevitsh Jepifanov. Hän oli parhaassa iässään, 37-vuotias muistaakseni, keskikokoinen kasvultaan, reipas liikkeiltään ja lyhyt puheiltaan, vaaleaverinen, huuli- ja leukaparta semmoinen kuin meidän puolen
- 52. merimiehillä usein näkee; oululaisesta perämiehestä tai kapteenista hän olisikin voinut käydä milloin hyvänsä. Kotoisin hän oli Usmanalta, jossa oli ollut perhettänsä tervehtimässä ja josta vaimonsa nyt saattoi häntä luostariin, kun hän oli paluumatkalla Vienaan (Arkangeliin). Siellä häntä nimittäin oli odottamassa oma pieni jahti, jolla hän kuljetti tavaroita Vienan ja Norjan välillä, vieden jauhoja, köysiä ynnä muuta tavaraa ja tuoden pääasiallisesti kaloja. Hänen jahtiinsa mahtui vientitavaraa 1,500 ruplan arvosta, ja kun hän itse ne osti ja möi, oli hänen liikkeensä, jota hän oli harjoittanut kymmenkunnan vuotta, hyvästi kannattava. Kuitenkin hän valitti sitä vaivalloiseksi ja uhkaili luopua siitä sekä ruveta lohta kuljettamaan Usmanalta Pietariin. Naimisissa hän oli ollut jo toistakymmentä vuotta, vaikka sekä hän että hänen vaimonsa kumpikin näyttivät niin nuorilta ja toisiinsa niin rakastuneilta, että ensin luulimme heidän viettävän avioliittonsa alkuviikkoja. Nämä tiedot saimme häneltä jälestäpäin sen viikon kuluessa, jonka oleskelimme toistemme seurassa ensin luostarissa ja sitten Vienassa. Vaikka ensi tuttavuutemme oli tehty vähän jyrkällä tavalla, tuli meistä näet kohta hyvät ystävät. Niinkuin jo olen osoittanut, ansaitsi hänen käytöksensä Kusovoin ulommaisessa saaressa pikemmin kiitosta kuin moitetta, ja mitä taas tulee tuohon olemiseen karpason kannen alla, joka kumppaliani oli vähän loukannut, saapi se tyydyttävän selityksen siitä, että hän — niinkuin myöhemmin kummaksemme kuulimme — ei kulkenutkaan veneessä meidän maksullamme, s.o. ilmaiseksi, niinkuin luulimme, vaan oli kuin olikin veneen isännälle maksanut kolme ruplaa. Osamiehenä veneen vuokraamisessa hän siis hyvällä syyllä saattoi itseänsä pitää. Mutta matkaan takaisin. Kun oli saaren sivutse ennätetty ja tultu aavalle merelle, alkoi sumu vähitellen kadota, tuuli lakkasi kohta
- 53. myöskin puhaltamasta, ja Vienanmeri lepäsi edessämme rasvatyynenä ja kirkkaana kuin peili. Suunta pantiin, ei suoraan luostaria kohti, joka on tuon kolmatta penik. pitkän saaren länsirannan keskikohdalla, vaan paljoa etelämmäksi, saaren etelänokkaa kohti, sillä pakovesi olisi muuten kantanut meidät merelle, niinkuin teki eräälle toiselle, perässämme tulevalle veneelle, joka oli kulkenut Kusovoin viimeisen saaren pohjoispuolitse. Kun puhun meren tyyneydestä, täytyy minun kuitenkin lisätä, että yksi osa siitä ei ollut tyyni: pitkin ulommaisen Kusovoin saaren syrjää jonkun virstan päässä rannasta näkyi aallokko ja kuului kohina ikäänkuin pahimmasta koskesta. Erinomaisen outoa oli katsella tätä kuohua keskellä muuten rasvatyyntä merta, emmekä voineet sitä ollenkaan ymmärtää; mutta miehet sitten selittivät, että siinä oli syvempi paikka ja että veden laskeminen synnytti siihen aallokon. Kiikuttuamme mekin vähän aikaa sen eteläpäässä jätimme kohta sen taaksemme ja lähestyimme sitten lähestymistämme yhä selvemmin "valottavaa" monasteria. Tuon tuostakin kuului Huotarin "pogrebiite, pogrebiite" eli "soudaltakaa, soudaltakaa, raukat" — raukka näyttää olevan hyväilysana naisille Karjalassa, sillä myötäänsä se miesten huulilta kaikui — mutta tämän kehoituksen hän lienee lausunut enemmän omaksi huviksensa, sillä soutajia ei sopinut laiskuudesta moittia. Kuta illemmäksi ja tyynemmäksi kävi, sitä tiheämpään näkyi ympärillämme, välistä etäämpää, välistä aivan likeltä, lumivalkoisia otuksia, jotka hetkeksi kohosivat vedenpinnasta ylös ja sitten vyörähtivät taas näkymättömiin; saattajamme sanoivat niitä "belugoiksi", ja ne arvatenkin olivat jonkunlaisia delfiinejä. Noin penikulman päässä luostarista on yksinäinen pieni kalliosaari meressä, Tuopin luoto nimeltään; sen ympärillä oli ikäänkuin sakeana savuna kimeästi kirkuvia tiiroja ja muita merilintuja. Jättäen oikealle kädelle, s.o. etelään päin, Jänissaaren (Sajatshij ostrovin),
- 54. jossa luostarin karjatalo on, kuljimme viimein kahden pienen luotosen välitse, joiden harjalle oli pystytetty suunnattoman korkeat puuristit, ja edessämme oli luostarin pieni satama, jonka laituriin pari laivaa ja lukematon joukko veneitä oli kiinnitetty ja jonka rannalla luostarin valkeat rakennukset kirkkaasti hohtivat alenevan auringon valossa. Luostari. Luostarin satamana on pieni, lännestä itään tunkeva merenlahti. Kivilaiturit on rakennettu sen ympärille, niin että se muodostaa länteen päin avonaisen neliön. Se on siksi syvä, että suuret höyrylaivat uivat sen perälle asti. Höyryt ja veneet lasketaan pohjanpuoliseen laituriin, sillä tällä puolen on kaksi suurta rakennusta matkustajia varten: ensimmäinen kivestä, kolmikerroksinen, kussakin kerroksessa 27 ikkunaa rinnatusten, rakennettu v. 1866, niin etäällä laiturista, että väliin syntyy torinlevyinen aukio, toinen puusta, kaksikerroksinen, edellisestä vähän pohjoisempana törmällä ja hiukan lyhyempi. Molempain etusivu on etelää kohti. Vastapäätä, sataman eteläpuolella, kauniin koivikon suojassa, on myöskin kaksikerroksinen puurakennus, johon ei kuitenkaan matkustajia laskettu. Pitkin sataman itäsyrjää pohjoisesta etelään kulki korkea kivimuuri kuin liimassa ainakin, jonka molemmissa päissä sekä keskessä oli pyöreät tornit ja jonka takaa luostarin valkeat kirkkorakennukset torneineen ja risteineen kohosivat. Rannalla seisoksivan lukuisan väkijoukon halki nousimme tuohon kiviseen "gostinnitsaan" eli majataloon, jonka läpi pitkä ja leveä korridori kulki länsipäästä itäpäähän, numeroidut ovet molemmin puolin kuin hotellissa ainakin. Keskellä korridoria oli vestibylit ja
- 55. uloskäytävät sekä etelään että pohjoiseen päin. Se munkki, joka tässä hotellissa oli isäntänä, tavattiin pitkän hakemisen perästä ylimmästä kerroksesta, ja monen mutkan jälkeen annettiin meille viimein eräs alakerran huone, johon paitsi meitä kahta myöskin toinen puoli matkakumppaleistamme turvautui, sillä kaikki paikat olivat matkustajia täynnä. Huoneemme oli suuri ja korkea, siinä oli kaksi ikkunaa satamaan päin, mutta hyvin yksinkertainen sisustus: kolme sänkyä, kussakin tuuman paksuinen patja ja kahta paksu päänalusta, pieni pöytä ja pari tuolia — siinä kaikki huonekalut. Yhdessä alanurkassa tietysti oli "bohumaaterin" pyhä kuva. Tämänkaltaisia näyttivät kaikki huoneet olevan alimmassa kerroksessa, jota käytetään yhteistä kansaa varten. Ylin kerros, johon kumppalini ja minut seuraavana päivänä puolisen jälkeen käskettiin ja johon muutimmekin, oli hiukan mukavammin sisustettu; keskikerros, parhaita vieraita varten, on kuitenkin vasta varustettu semmoisilla tarpeellisilla huonekaluilla kuin pesukaapeilla, sohvilla y.m. Tavallisesta ravintolasta tämä gostinnitsa erosi siinä suhteessa, että siellä ei voinut tilata mitään ravintoaineita paitsi kiehuvalla vedellä täytetyitä samovaareja teetä varten. Eväät piti siis itsekullakin olla muassa — jollei halunnut määrätyillä ajoilla käydä itse luostarissa yhteisessä pöydässä syömässä. Näitä aterioita jokainen saa käyttää hyväkseen kolmen vuorokauden ajan maksuttomasti. Samovaareja, joita täällä näkyi olevan runsaasti, kiehui kohta yksi meidänkin huoneemme pöydällä, ja lasi teetä sekä sen kera kappale Usmanan lohesta tehtyä kalakukkoa maistui sangen hyvältä, päivän merellä oltua. Mutta syötyä kumppali B. ja minä pakenimme ulos luostariseutua katselemaan.
- 56. Ensimmäinen mikä luostarin rannalla vetää vieraan silmät ja varsinkin korvat puoleensa, on ääretön joukko kalalokkeja (tshaikkoja), jotka joka askeleella pyörivät kulkijan jaloissa ja aukoen suuria suitansa kimeällä äänellä huutelevat ruokaa. Ne ovat tulleet aivan kesyiksi sen takia, että luostarisaarella kaikki otukset ovat rauhoitetut ja että pyhiinvaeltajat, arvatenkin pitäen näitä lintuja jonkunlaisina paikan pikku-isäntinä, kilvan syöttelevät ja ruokkivat niitä. Kauhea on se parku ja melu, jonka tämä ehkä tuhatmäärään nouseva lintuparvi saapi aikaan, niin että korvat ovat haljeta, ennenkuin siihen tottuu. Se ajatus, että hengiltä älköön mitään elävää saarella otettako, on kyllä hyvin kaunis, mutta oudolta aluksi tuntuu, että paikka, jonka juuri on määrä tarjota hiljaisuutta ja rauhaa maailman levottomuuteen kyllästyneelle mielelle, noitten lintujen elämöimisessä esittää kaikkea muuta kuin äänettömyyden ja sovinnollisuuden kuvaa. Puolipäiväsaarnaa seuraavana päivänä luostarin tuomiokirkossa pidettäessä oli muuan lokki lentänyt kirkon katolle ja korotti siellä odottamatta kesken rukouksia kirkuvan äänensä, joka kuului sitä paremmin, kun katossa oli joku aukko. Venäläisten tavallisella välinpitämättömyydellä jumalanpalveluksessa ei kirkkoväki kuitenkaan näyttänyt tätä ollenkaan huomaavan. Pitkin sataman itäsyrjää, parin kolmen kadunleveyden päässä laiturista, kulkee niinkuin jo sanoin kivimuuri. Se on osa eli syrjä siitä suojelusmuurista, joka ympäröi koko luostaria, muodostaen epäsäännöllisen viisikulmion. Tämä suojelusmuuri on 5-6, paikoin ehkä 7:kin syltä korkea ja monta syltä paksu, rakennettu äärettömän suurista, hakkaamattomista munakivistä, ylhäältä ylt'ympäri varustettu ampumarei'illä sekä lisäksi vahvistettu torneilla, joita on yhteensä 8 tai 10. Sisäpuolella kulkee ampumareikien tasalla katettu käytävä muurien puolustajia varten. Muurien läpi johtaa eri kohdilla viisi kuusi porttiholvia, jotka yöksi salvataan kiinni vankoilla
- 57. raudoitetuilla porteilla. Luostari on siis luja linna, jota varsinkin siihen aikaan eli 300 vuotta takaperin, jolloin sen muurit rakennettiin ja jolloin ampumakoneet eivät olleet niin pitkälle kehittyneet kuin nykyään, oli melkein mahdoton vihollisen valloittaa väkirynnäköllä. Itäpuolella luostaria on näet lisäksi lampi eli pieni järvi, Pyhä järvi (svjätoje osero), joka sitä puolta suojelee, etelään päin taas lammesta satamaan juokseva syvä oja; pohjan puoli luostaria on rakennettu mäelle, ja muurien juuresta alkava rinne on täällä jonkunlaisena puolustuksen apuna. Tätä puolta muuten näytään pidetyn vaarallisimpana, koska muurit ovat siellä korkeimmat ja tornit tiheimmässä. Mutta se onkin lyhyin, sillä luostarin pituussuunta on pohjoisesta etelään. Satamaan päin tuleva osa muuria on pisin, lähes yhtä pitkä kuin koko luostari; sen molemmissa päissä muuri tekee hyvin jyrkät käänteet. Askelten mukaan lukien sen pitäisi olla noin 150 syltä, siis 1/4 virstaa; koko luostarin ympärys luetaan runsaaksi virstaksi. Lehtipuita on istutettu sitä pitkin verhoamaan kivien alastomuutta. Siinä on kaksi porttia: toinen gostinnitsan kohdalla, toinen etelämpänä. Jälkimmäinen, jota sanotaan "pyhäksi", on pääportti, josta kuljetaan suoraan tuomiokirkkoon; sen yläpuolella muurissa on freskomaalauksia. Pyhästä järvestä juoksevan ojan poikki vei monta siltaa, joista alimman korvissa oli kaksi pienoista obeliskia. Kun niitä katsellaksemme lähestyimme siltaa, huomasimme sen ja vähän ylempänä olevan toisen sillan välissä — laivatokan, jommoista ei meidän maassa ole muualla kuin Helsingissä! Kummastuksemme ei ollut vähäinen, sillä semmoista emme tosiaankaan olleet täällä odottaneet näkevämme, — vaikka tässä suhteessa jälestäpäin saimme syytä vielä suurempaan kummastukseen, kun kuulimme,
- 58. että tuo laiturin kupeella oleva höyrylaivakin oli luostarissa tehty. Olivatko sillan korvassa seisovat obeliskit tokan rakentamisen vai jonkun muun tapauksen muistoksi pystytetyt, en tiedä sanoa, kyllähän niissä taisi olla joku slavonialainen kirjoitus, mutta siitä emme saaneet selvää. Lounaisosassa luostarimuuria, joka tokan kierrettyämme oli edessämme, näkyi melkeinpä lukematon joukko päänkokoisia mustia täpliä sininen rengas ympärillänsä: ne olivat muistoja englantilaisten sotataivain käynnistä näillä vesillä 1854 ja luostarin silloin tapahtuneesta pommituksesta. Ei kuitenkaan voinut huomata, että pommit olisivat muuriin mitään vahinkoa tehneet. Sitten jatkoimme kulkua ympäri luostaria pitkin järven rantaa; rannan ja muurin välissä kulkee näet kadun levyinen tie. Luostarin pohjoispäässä näkyi korkealta muurien yläpuolelta ristikkoikkunat — siellä siis oli vankihuoneita. Järven pohjoisessa päässä vähän matkaa muurista oli ryhmä kaikenlaisia rakennuksia. Kiertokulkumme jälkeen kävelimme vielä hetken aikaa majatalon ja luostarin väliin tehdyllä lankkukäytävällä. Kun katseli tuota uutta komeaa kivirakennusta, sen edustalla olevaa toria ja laituriin kiinnitettyä höyrylaivaa, olisi voinut luulla seisovansa kauppatorilla Helsingissä Seurahuoneen edustalla. Mutta toinen puoli näköalaa hävitti tämän kuvitelman: tuo vankka luostarimuuri ja sen takaa kiiltävät valkeat seinät, vihreät katot ja lukemattomat torninhuiput risteinensä muistuttivat kohta, että muualla oltiin. Muukalaisuutta johtivat mieleen myöskin rannallavilisevät väkilaumat, niin kauan kuin heidän vieras puheensa kaikui korvissamme, niin että vähän ristiriitaisin tuntein nautimme kesäyön suloisuutta. Mutta kun rannallaolijat vähitellen kukin sortuivat majoihinsa ja nuo kalalokitkin herkesivät huutamasta, alkoi yön hiljaisuudessa pohjanpuolisesta lehdikosta kuulua käen heleä kukunta, ja tämän lempilintumme tuttujen sävelten turvissa mekin siirryimme makuuhuoneeseemme.
- 59. Seuraava päivä, 11 p. heinäk., oli Pietari-Paavalin suuri juhlapäivä, joka vastaa meidän juhannusta. Jo neljältä aamulla sanottiin ensimmäisen jumalanpalveluksen alkaneen, mutta matkatoverimme eivät olleet kirkkoon mennessään tahtoneet meitä herättää, kun ei siitä ollut sovittu, josta olimme heille hyvin kiitolliset. Kun 7:n korvissa heräsimme, oli huone kummaksemme tyhjä, mutta vähitellen ilmestyivät kumppalimme toinen toisensa perästä ja viimein myöskin tuo unum necessarium, samovaari. Pesemisen suhteen oli iso hankaluus sen puolesta, että vettä ei voinut saada huoneeseensa, vaan pesu oli toimitettava pohjoisen uloskäytävän porstuassa. Siellä oli kummallakin seinällä kyynärän korkeudella lattiasta leveät rännit ja niiden yläpuolella puolisen tusinaa päänkokoisia messinkipalloja, joiden pohjasta ponnella varustettu korttelin pituinen hieno messinkitanko riippui. Kun tätä lykkäsi ylös palloon, juoksi reiästä vettä kahden puolen tankoa alas. Epämukavaa laitos, mutta "mushikat" ehkä pitävät sitä hyvinkin hyvänä. Juomavettä ei myöskään ollut huoneissa, mutta korridorissa oli janon sammuttamiseksi suuret astiat täynnä sahtia eli kaljaa, "kvassia", joka oli luostarin suuressa panimossa tehtyä ja erinomaisen hyvänmakuista. Päivän alkuvalmistuksista päästyämme kiirehdimme mekin luostarin muurien sisäpuolelle, jossa jo pariinkiin toviin lienee jumalanpalvelusta pidetty. Kun pääportin kautta aikoo sisään, on ensin kuljettava kivipaasilla lasketun holvin läpi, joka on 15 syltä pitkä. Sen katossa riippuu kaksi laivanmallia, jotka Pietari Suuri on lahjoittanut luostarissa käydessään, toisen 7 p. heinäk. 1694, toisen 10 p. elok. 1702; ne ovat vanhaa hollantilaista mallia, perä korkea ja siinä ikkunat. Kun holvista astuu kartanolle, on vastassa Kristuksen kirkastuksen
- 60. tuomiokirkko (Preobrashenskij sobor), jonka perustukset "iguumen" eli apotti Filip laski v. 1558 ja jossa luostarin ensimmäisten perustajain, Sauvatin ja Sosiman, komeat ruumisarkut säilytetään. Kirkon portaille johtaa leveä, 30-40 syltä pitkä käytävä kartanon poikki, joka on pensaita ja kukkasarkoja aivan täynnä, niin että se näyttää mitä kauneimmalta kasvitarhalta. Kirkastuksen kirkko on eteläosa laajasta rakennussarjasta, joka ulottuu kartanon pohjoispuoleen asti. Paitsi erinäisiä muita huoneita, niinkuin esim. kirjastoa, on tässä sarjassa kolme kirkkoa: lähinnä Kirkastuksen kirkkoa, vähän edempänä, niin että se yhtyy itämuuriin, eräs kirkko, jonka katto on sininen ja kultatähdillä koristettu (kaikki muut katot ovat vihreitä) ja jonka Dixon (kirjassaan "Free Russia") sanoo rakennetun erään englantilaisen laivaston karkoittamisen muistoksi, mutta jota toinen niistä kirjoista, jotka luostarissa ostin, nimittää piispa Filipin kirkoksi; senjälkeen piispa Nikolain kirkko ja viimein pohjoisimpana Neitsyt Maarian taivaaseenastumisen kirkko (Uspenskij sobor), jota käytetään refektoriona eli ruokasalina ja jota talvella saatetaan lämmittää. Senkin rakennutti jo mainittu apotti Filip vv. 1552-57 ja se lienee vanhin luostarin nykyisistä rakennuksista. Nikolain kirkon vieressä on kellotapuli. Pohjoispuolella tätä rakennussarjaa on avoin piha, jonka luoteispäässä on gostinnitsan kohdalle tuleva portti ja jonka pohjoissyrjää reunustaviin korkeihin rakennuksiin tehdyn toisen portin eli holvin kautta tullaan korkeiden rakennusten välissä olevalle pienelle pihalle: se on vankien piha, ja nuo korkeat rakennukset ovat vankihuoneita. Lukematon joukko kyyhkysiä asuu täällä vankien parissa.[12] Paitsi jo mainittuja neljää kirkkoa oli vielä yksi, erillänsä toisista ja etelämuurissa kiinni. Se on varmaan se Maarian ilmestyksen kirkko, joka rakennettiin kohta sen jälkeen kuin muurit luostarin ympäri oli saatu valmiiksi (v. 1594).
- 61. Pitkin muurien sisäkylkeä kulkevat ylt'ympäri kaksi- tai kolmikerroksiset kivirakennukset. Niissä on luostariväen asunto- ja työ- ynnä muut huoneet. Pääportin yläpuolella on luostarin esimiehen, arkkimandriitin, asuinkerta, ja ylhäällä ilmassa kulkee siitä katettu käytävä pihan ylitse kirkkorakennussarjaan. Itämuuriin liittyvät huonerakennukset ja usein mainittu kirkkosarja ovat niin lähellä toisiansa, että ainoastaan katu mahtuu väliin, ja senkin yli ulottuvat yhdessä tai parissa paikassa kirkkorakennukset, niin että syntyy suuria holveja. Kirkkosarjan keskikohdalla kulkee myöskin tämän kadun ja varsinaisen luostarikartanon välillä mutkitteleva holvi, jonka molemmilla puolilla on kellarintapaisia hautakammioita, kolkkoja, kylmiä ja sydänpäivälläkin niin pimeitä, että niissä palavaa lamppua kyllä tarvitaan. Eräässä tämmöisessä on (luostarin kolmannen alkuperustajan?) Germanin muumio, jota ristikkoikkunan läpi hyvästi saattaa katsella. Kammion ovi oli kiinni, mutta vastapäätä, puoleksi maan alla, oli toinen hautakammio, jonka avonaisesta ovesta astuimme sisään. Kuka tässä lepää, en jaksa muistaa, kun luostarin historia silloin vielä oli minulle aivan tuntematon; kenties Theofan, kenties Nahum; väristen siitä kohta pakenimme. Holvikäytävän syrjillä oli myöskin hautoja ja komeita ruumisarkkuja, yksi niistä sisältäen piispa Filipin jäännökset, jollen väärin muista. Kellotapulin edustalla on alhaalla maassa kello, jota soitetaan ylhäältä tapulista juoksevan nuoran avulla. Sen ympärille on muistoksi englantilaisten pommituksesta pystytetty pyramiideja eheistä ja särkyneistä pommeista, niin ikään pari pientä tykkiä, joita luostarin puolustukseksi käytettiin.
- 62. Tämän näköinen on luostarikartano pääpiirteiltänsä. Astukaamme jo kirkkoihin. Se vaikutus, minkä tuomiokirkko meihin teki, oli kerrassaan masentava. Seinät ovat ylt'ympärinsä täynnä taiteellisesti tehtyjä maalauksia taivaansinisellä pohjalla, jotka esittävät tapauksia raamatusta, — erittäin jäi mieleeni lastenmurha Betlehemissä, vuorisaarna, vapahtaja pesemässä opetuslasten jalkoja ja myrsky merellä. Lähes keskellä kirkkoa on kupulakea kannattamassa kaksi neliskulmaista, sylen paksuista pylvästä, joitten kulmiin sovitetut hennot, lehdentapaisilla koristuksilla peitetyt kiertyvät pilarit varsinkin vetivät silmää puoleensa. Niiden edustalla oli miehenkorkuiset, hopealla päällystetyt kynttiläjalat. Ikonostaasi eli se väliseinä, joka erottaa pyhimmän osan kirkosta, oli lattiasta huimaavan korkeaan kattoon asti täynnä pyhäin miesten muotokuvia, ja keskellä paistoi suunnattoman suuri Kristuksen kuva hopeaisessa puvussa. (Venäläiset usein verhoavat Kristuksen ja Neitsyt Maarian kuvat hopea- tai kultavaatteella, jättäen aukon ainoastaan kasvoja ja käsiä varten, joka outo tapa lienee saanut alkunsa siitä, että pyhäinkuvain otsaan ensin ripustettiin kruununtapaiset diadeemit.) Kultaa ja hopeaa hohtivat kaikki paikat kirkossa, missä ei maalauksia näkynyt, niin että silmiä tahtoi huikaista. Tuomiokirkon länsi- ja pohjoispuolella oli salintapaiset suuret etuhuoneet, joiden seinät niin ikään ovat maalauksilla täytetyt ja lisäksi penkeillä varustetut. Pohjanpuolisen etuhuoneen itäpäässä on ovi tuohon sinikattoiseen kirkkoon, joka on yhtä ylellisen komeasti sisustettu kuin tuomiokirkko. Etuhuoneitten luoteisesta kulmasta lähtee pitkä galleria, jolla apotti Isidor noin v. 1600 yhdisti kaikki kirkkorakennukset toisiinsa. Sen itäisellä puolella on järjestään
- 63. käytävät Nikolain kirkkoon, kellotapuliin, kirjastoon ynnä muihin huoneisiin ja viimein refektorioon, jonka edustalla se levenee salintapaiseksi porstuaksi. Seinämaalaukset tässä galleriassa ovat toista laatua kuin muualla: ne kuvaavat ensin kaikenlaisia ihmeitä, joita tapahtui luostarin ensimmäisille perustajille, ja esittävät sitten jos jonkinmoisia hirvittäviä kuvia paholaisesta ja helvetistä. Oppimaton, yksinkertaisesti uskovainen kansa epäilemättä pyhällä kauhistuksella niitä katselee. Nykyisen kellotapulin, johon noustaan hyvin kaitaisia portaita, rakensi, jollen erehdy, arkkimandriitti Dosifei sata vuotta sitten. Siinä on yhteensä 29 isompaa ja pienempää kelloa: suurin painaa 1,100 puutaa, siis lähes 2,000 leiviskää. Kaksi miestä tarvittiin sen soittamiseen, s.o. saadakseen kielen koskemaan laitaan. Ilman tärinä oli pienessä huoneessa niin väkevä, että jonkun minuutin päästä täytyi lähteä pakoon sieltä. Tyynellä ilmalla sanotaan kellon äänen kuuluvan Kemiin asti. Refektorio, talvikirkko, on avara huone ja samoin kuin kaikki muutkin paikat kaunistettu maalauksilla ja muilla pyhillä koristeilla. Dixon, jo mainitussa kirjassaan, sanoo refektorioksi käytetyn tuomiokirkon alla olevaa huonetta. Siinä emme käyneet. Vielä on lopuksi lisättävä, että tuon tuostakin seinillä, pylväissä tai jossakin muurin nurkassa seisoi komea Kristuksen tai Neitsyt Maarian kuva, jokainen luultavasti "Ihmeitätekevä". Me pakanat emme niihin tulleet kiinnittäneeksi erityistä huomiota, koska noissa maalauksissa oli enemmän katsomista, mutta "oikeauskoiset" varmaan tekevät tarkan luettelon kaikista niistä. Väkeä vilisi tänä päivänä kartanot, käytävät ja kirkot aivan täynnä niinkuin parhaimmilla markkinoilla. Omia asukkaita kuulimme
- 64. luostarissa olevan kaikkiaan 500:n ja 1,000:n hengen välillä, mutta pyhiinvaeltajia oli nyt arviolta kahdenvertainen määrä, s.o. toista tuhatta henkeä. Joku osa niistä oli kotoisin rannikkoseuduista, Karjalasta ja pohjois-Venäjältä, ja olivat omilla veneillään tänne purjehtineet, mutta suurin osa oli tullut höyrylaivoilla Vienasta ja olivat kotoisin sisä-Venäjältä. Venäläisten samoin kuin muhamettilaisten uskontoon kuuluu nimittäin vaeltaminen pyhiin paikkoihin, joita Venäjällä on koko joukko ja joista Solovetsin luostari on mainioimpia. Kun niissä on jonkun "ihmeitätekevän" pyhänkuvan edessä tehty ristinmerkkejä ja lyöty otsaa lattiaan, sitten suudeltu pyhimysten ruumisarkkuja ja kuunneltu messua sekä ostettu vahakynttilöitä palamaan jonkun kuvan eteen, on muka tehty Jumalalle erittäin otollinen työ, josta autuuden porttia ainakin raolle aukaistaan. Että tämmöiset pyhiinvaeltajat eli "bohomoltsit" (rukoilijat) melkein yksinomaan ovat alhaista, sivistymätöntä kansaa, on itsestään ymmärrettävää; sentähden mekin koko tässä ihmislaumassa tuskin näimme kymmentä säätyhenkilöä. Että moni heistä myöskin on laiskuri, joka kernaammin maleksii maita mantereita kuin tekee työtä, voinee varmasti väittää, vaikka pääosa lieneekin vilpittömästä uskonhartaudesta liikkeelle lähtenyt.[13] Puolipäiväsaarnan eli messun toimituksessa oli kymmenkunta ylhäistä pappia osallisena, kaikilla sentapaiset kultalankakankaasta tehdyt kaavut yllään kuin meidän piispoilla kirkollisissa juhlatilaisuuksissa. He seisoivat piirissä keskellä lattiaa; alimpana joukossa, selin ovea ja päin alttaria kohti, oli pitkäpartainen, kunnioitettavan näköinen vanhus, jota sanottiin väliaikaiseksi arkkimandriitiksi. Varsinaista arkkimandriittia ei näet luostarissa ollut, sittenkuin entinen oli joko kuollut tai nimitetty muuhun virkaan. Pitkin ovenpuolista seinää istui erinäisillä istuimilla puolisensataa munkkia. Kaikki meno näytti erittäin juhlalliselta, ja kun laulukunta
- 65. papin saarnaan tuontuostakin kajahutti "hospodi pomiilui" eli "herra armahda", kumartuivat sanankuulijat ja tekivät ristinmerkin suurimmalla nöyryydellä ja katuvaisen näköisinä; vieläpä useat heittäytyivät polvillensakin lattialle, johon painoivat otsansa monta kertaa. Kumarruksensa ja ristinmerkkinsä sekä otsanlyönnin lattiaan, jotka näyttävät olevan pääasiana venäjänuskoisten hartaudessa, he muuten toimittavat aina hyvin totisesti, niin että vieras, vaikka kummastellen sitä katselee, ei voi siitä pilkkaa tehdä. Edellämainitussa galleriassa ennen jumalanpalvelusta kulkiessani sattui silmäni vanhanpuoliseen mieheen, joka muutaman nurkan takana laskeusi kasvoillensa lattialle. Kohdalle tultuani huomasin seinässä pyhänkuvan, jolle tämä kunnia osoitettiin. Nojaten selkääni seinää vasten käytävän toisella puolella jäin ihmetellen siihen seisomaan sylen päähän miehestä, katselemaan hänen temppujansa. Noustuaan pystyyn hän teki ristinmerkin, kumartui syvään ja laskeusi sitten taas ensin toiselle polvelle hyvin verkalleen ja varovasti sekä löi viimein otsansa lattiaan. Tämän hän minun läsnäolostani vähintäkään huolimatta uudisti noin kymmenen kertaa, montako sitten jo varemmin lie suorittanut! Niin oudolta kuin tämä meno ensin näyttikin, en kuitenkaan voinut suutani vetää nauruun, koska miehen kasvoissa kuvastui vilpitön hurskaus, ja kun hän viimein tyytyväisenä lähti astumaan edelleen, jäin paikalleni syvästi vakuutettuna hänen jumalisuudestansa. Usealle kuitenkin ristinmerkki alituisen, joka päivä kymmeniä kertoja tapahtuvan tekemisen kautta lienee muuttunut paljaaksi ulkonaiseksi muodoksi, johon ei sielu eli henki ota suurta osaa. Niinpä vielä siinä seisoessani näin toisen bohomoltsin seisahtuvan kuvan eteen. Hän ei heittäytynyt maahan, teki ainoastaan
- 66. ristinmerkin ja kumartui ehkä kolme kertaa, mutta kääntyessään matkaansa jatkamaan — haukotteli hyvin hartaasti! Toiset taas osoitettuaan kunnioitustansa niistivät nenäänsä (sormilla), sylkäisivät tai töllistivät, ylimalkaan käyttäytyivät niinkuin sivistymätön meikäläinen maallisissa jokapäiväisissä toimissaan. Paikalta lähtiessäni siis ensimmäinen kunnioituksen tunteeni asianomaisten hartautta kohtaan oli melkoisesti laimentunut. Tuo ristinmerkin teko on aatteeltaan jotakin kaunista, koska siten ikäänkuin painetaan Jumalan siunaus hankkeelle ja toimelle; ennen uskonpuhdistusta meidänkin esi-isämme olivat siihen tottuneet. Mutta kun sitä liian paljon tai muuten sopimattomasti käytetään, menettää se vaikutuksensa. Erinomaisen huvittavaa oli esim. iltapäiväjumalanpalveluksessa katsella muuatta pientä, virkkusilmäistä poikaa, kenties 9-vuotiasta, joka kulki vanhemman miehen, arvatenkin isänsä seurassa. Niin ahkerasti kuin hän ei varmaan yksikään muu kirkkomies ristinmerkkiä tehnyt; tuskin hetkenkään loma-aikaa hän malttoi pitää. Ja ennenkuin kumartui eteenpäin, hän aina ensin notkisti selkänsä taaksepäin, vauhtia saadaksensa, niin että ruumis huojui yhtenä luokkana edestakaisin. Meidän voimistelu kouluissa semmoinen ruumis epäilemättä ottaisi ensi palkinnon. Itse merkki voidaan muuten tehdä eri tavalla, sievästi tai rumemmasti. Talonpoikainen kansa enimmäkseen muodostaa ilmaan niin suuren ristin kuin mahdollista koskettaen sormillaan ensin otsaan ja vatsanpohjaan, sitten oikeasta olkapäästä vasempaan. Tuo v.t. arkkimandriitti kirkossa teki merkin hienommasti: hän hiukan vain vilahutti kättä rinnallansa, niin että enemmän aavisti kuin näki ristinmerkin tehdyksi.
- 67. Sormienpito merkkiä tehdessä on pääerotuksia n.s. starovertsien eli vanhauskoisten ja valtiokirkon kannattajain välillä. Edelliset painavat peukalon nimetöntä sormea vastaan ja pitävät etu- ja keskisormet suorina; jälkimmäiset painavat peukalon etusormea vastaan. Messun jälkeen B. ja minä palasimme majapaikkaan, sittenkuin sivumennen muutamasta myymälästä pääportin vieressä (sisäpuolella muuria) olimme ostaneet rihman semmoisia valkeita rinkeleitä, joita karjalaiset markkinoillamme kaupittelevat. Vaikka oli muka pyhäpäivä, tehtiin tässä vilkasta kauppaa, puoti oli väkeä aivan täynnä Olivatko nuo rinkelit täällä tehtyjä, en tullut tiedustelleeksi, mutta se on hyvin luultavaa, sillä luostariväki valmistaa kaikki tarpeensa itse. Karjalassa niitä ei tehdä, vaan ne tuodaan Äänisjärven tienoilta sinne; joka paikassa niitä kuitenkin saapi ostaa maakauppiailta, ja niitä myydään painon mukaan, jolloin punnitessa käytetään vanhanaikaista puntaria. Vedestä ja nisujauhoista tehtyinä niiden maku ei juuri ole mikään kehuttava, mutta makeisiin tottumattomat karjalaiset niitä yksin suin ylistävät erinomaisiksi. Rihma maksaa 15-20 kopeekkaa. Majapaikka oli taaskin melkein tyhjä, sillä suurin osa kumppaleistamme oli messun jälkeen mennyt syömään luostarin yhteiseen pöytään. Me olimme väentungoksessa joutuneet erillemme heistä emmekä siis tienneet tehdä heille seuraa. Jepifanovin toimesta muistaakseni kuitenkin kohta saimme teekeittimen eteemme, ja sen sekä evästemme avulla nälkä oli äkkiä poistettu. Mutta keittoruuan tarve alkoi vähitellen käydä tuntuvaksi, ja sekä sitä että myöskin uteliaisuuttamme samalla tyydyttääksemme päätimme illalla mekin mennä luostaripöytään.