Distributed application usecase on docker

Copyright © 2016 NTT DATA Corporation
15, July 2016
Hiroshi Miura
Systems Engineering Headquarters, NTT DATA Corporation
Distributed application use case on container clusters -
automated operations with Hinemos and Docker

2Copyright © 2015 NTT DATA CorporationCopyright © 2016 NTT DATA Corporation
Self introduction
• Senior IT-Specialist for Platform
on NTT DATA Corporation.
• Lead Platform Engineering Services
using OSS/Linux and related
technologies
• Github/Twitter: @Miurahr
• Contributions:
– Some Linux drivers for laptop, CPU,
power management etc.
– Tools such as Vagrant-kvm
– OpenStack Swift
– OmegaT Translation tool
– Etc….

Evaluation project with Docker in enterprise application.
Service Information in NTT DATA web page in 2015 fall.

Evaluation project with Docker in enterprise application.
Service Information in NTT DATA web page in 2015 fall.
Docker Enterprise use case
Start PoC
We now resulted it is useful and go forward for production!!
After a half year

Why Docker/ container
From Enterprise Docker/Container BoF<on LinuxCon/ContainerCon 2015 Tokyo>
Discussed about barriers and solutions for enterprise use case.

Why Docker/ container
From Enterprise Docker/Container BoF<on LinuxCon/ContainerCon 2015 Tokyo>
Discussed about barriers and solutions for enterprise use case.
→ Agrees to share use cases.

2. lightweight
Docker from enterprise application perspective
1. Portability
3. History management 4. API: Docker Hub ecosystem
evaluation Private cloud
container
Apps A Update
Apps A
Apps A
Std. libs
CentOS6 Base
Apps B
Library B
Std. libs
CentOS6 Base
Share base images
C
h
a
n
g
e
l
a
y
e
r
Source
Container
definition
Image
management
Automated
test and
deploy
Automated
launch
Automated
launch
Server
OS
application
Virtual server
OS
application
コンテナ
Hypervisorapplication
Low
overhead
Hight
overhead

General use case for Docker/Container
Popular use case： Web application development/DevOps
• Portable application environment, wrap environment difference
• Easy deploy and accelerate delivery.
• Integrated environment for integration test and real service.
Dev
VM
Service
Bare metal
Test
container
test
feedba
ck
release
commit deploy deploy

Docker Use case
Media search system with finger print
• Massive images/videos/sounds search system
• High speed and precise detection of snippets
• Use finger prints of sounds and images
• SaaS service for media search and detection at Media business division
at NTT Data.
• Applied NTT Research and Development results.
RMS Engine
contents 特徴データ
・・・
Hit！
1
2
3
4
N
特徴データ
Search rapidly
Query（目的信号） Database（蓄積信号）
特徴データ抽出
関連情報付与
Sound
Video
Result(Music tile ・CD Identifier・Movie title etc.)
照合
# Service Application
1 Full music report
service
Watch and count
music utilization in
broadcast, and report
for copyright
accounting.
2 Contents
monitoring
Watch illegally
uploaded contents on
major sites world wide
3 Contents filtering Detect illegal upload
on movie sharing sites.
4 Several services CM combined service
etc.

Docker Use case
Features
• Distributed processing architecture with 10 – over 100 servers
• Use heavily hardware resources such as CPU and memory.
• Multiple application for multiple services and customers.
Finger print database
Distributed db
Service A cluster
server
AP a
serve
r
AP a
サーバ
AP a
・・・
サーバ
AP a
Services B cluster
サーバ
AP b
サーバ
AP b
・・・
サーバ
AP b
・・・
Distributed process for
search data in each
servers.

Technical findings
5
5.5
6
6.5
7
7.5
8
100 300 500 700 1000
TAT
Test Cases #
Average TAT
コンテナ
ハード
Application performance between bare metal and container
- 1.0 sec

Technical findings
• Key difference
– Bare metal environment: CentOS 6.x
– Docker host environment: CentOS 7.x (guest libraries : 6.x)
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
CPUUsage
CPU cores
CPU load %
物理
コンテナ

Docker use case
Challenges
To archive performance and deliverable time, operator need
to work on complex data distribution management.
VMware and/or OpenStack? Not good for “High resource demand”
Docker/Container? Both flexibility and low overhead
Different dependency for system on each
applications.
=>Impossible to integrate environment
=> High management cost.
Balancing server spec and
process load
Difficult to restart with fallback servers.
⇒ standby with whole system deployment
=> Duplex investment!!
Service A cluster
Server
AP a
Serv
AP a
サーバ
AP a ・・・
サーバ
AP
サーバ
AP a
サーバ
AP a
？？
Service B cluster
サーバ
AP b
サーバ
AP b
・・・
サーバ
AP b

Dockerの活用事例ご紹介
Automated distribution of App/Data by Docker Container
Deploy Docker containers on servers for each data management unit.
⇒ Automated deployment. Multiple application deployment with
common servers.
Server Server Server
・・・
APa APa APa APa APa APa
・・・
APa APa APa
Server
APa APa
Server Server
APb APb APb APb APb
Server
APb APb
APa APa APa APb APb APb
・・・・・・
Container
registory
Deploy on server with
load balanced way.
Service a Service b
Single server
cluster for
multiple
applications

N
A
N
A
Docker use case
Rebalance containers when hardware corruption
When detecting hardware defeat, rebalance containers according to
resource status. ⇒ Reduce cost and down time by automation
・・・AP AP AP AP AP AP AP AP
Server
AP AP
Server
AP AP
Server
AP AP
Broken
・・・AP AP AP AP AP AP AP AP AP
Server
AP AP
Server
AP AP AP
Server
AP AP
Replace affected
containers

Docker Use case
Key points for enterprise
Docker + Enterprise => Operation Improvement
• Automate rebalance keeps app performance
and reduces PF cost
• Squeeze hardware resources by precise
container placement
①Automated operations
• Reduce downtime and operational cost by
automated operations by trigger from breakage
detection
②Automated fallback on breakage

Docker Use case
Key points
Docker + Enterprise => Operation Improvement
• Wrap a PF difference for applications inside
container image.
• Consolidate operation procedure and platforms
③Consolidation of application PFs
• Consul name-resolver provide communication
base for dynamic PF
• Archive an operational improvement only with
platform change
④Non Application modification

Missing piece
WHAT is a missing piece to deploy

Docker use case
communication using consul name resolver
Communicate using name resolver by consul.
⇒ Containerize application w/o Modifications
Search
001
Search
003
10.0.1.2
Search001 10.0.1.2
Search002 10.0.2.2
Search003 10.0.2.3
Search004 10.0.3.2 ⇒ 10.0.4.2
Add 001
Register itself
On startup
10.0.2.3
Search
002
10.0.2.2
Resolve Search001
⇒10.0.1.2
communicate
Search
004
10.0.3.2
Search
004
10.0.4.2
Broken replace
Update registration
Update 004
consul
10.0.0.0/8

Docker User case
Automated operations
• Hinemos works as delegate of operators:
• container replacement, restart it and update provisioning db.
• Breakage detection, fallback
Server
r
Server Server Server Server Server
Container
Contain
er
Container
COntain
er
COntain
er
コンテナ
コンテナコンテナ
AP AP
AP AP
AP
AP APAP AP
コンテナ
コンテナ
コンテナ
AP AP AP
AP AP AP AP AP AP
Breakage
detect
コンテナ
AP
コンテナ
AP
コンテナ
AP
Replacement
Replacement of
container,
then monitor it
Detect breakage and
replace containers
placement・
Monitor

Hinemos
System monitoring and management application
for Enterprise applications.
 Born in Japan, home of quality crazy users
 Born as Open Source Software from first bit
 Simple UI for trivial use cases
 Multiple environmental support
 Private systems, Public cloud and Containers
 Linux, Windows
 PostgreSQL, MySQL, Oracle
 Job management capability, complex job executions

(1)Docker deployment automation
[Preexist Docker management products and services]
・From Image repository to orchestration for infrastructure layer
・No tools to monitor both application and infrastructure layer, and job
control
Docker Registorygitリポジトリ
commit ビルド
オンプレミス環境仮想化環境クラウド環境
コンテナ動作インフラ
デプロイ
オーケストレーション
クラスタ
VCS Image Management
CI Tool
dev
Docker
Operation tool
モニタリング
Job control
Monitor
both container and host
Missing place

Docker Job management: container operations
• Operates Start/Stop of Docker containers, by GUI and scheduled way.
• Remote and Agent less operation of Docker Containers.
• Integrate cloud management and Container operations.
Launch container
Restart container
Stop container
Remove container
Delete EC2
instance
Launch EC2
instance
Stop EC2 instance
Create EC2
instance
Backup EC2
instance
Hinemos Container management Hinemos Cloud management

Docker Job management
• Support job management on-premises, private/public cloud
and container environment.
• Schedule execution for container internal commands.
• Agent less execution for container commands.
✔
✔
▲
Remote
＆
Agent less
Job controls

Docker Job management
Hinemos can call docker API natively.
Two job execution ways
Hinemos(Docker API)Old Hinemos way
CLI
Proce
ss X
Process in
Container A
Launch
nsenter
(similar to
call ssh)
CLI
Proce
ss X
Process in
container A
Launch
Docker API(exec)
Container A
cgroups
Other
cgroups
Container A
cgroups

(2)Docker Container configuration management
• Push button management only with specify Docker host.
• Automated update follows life cycle of Docker Containers
• Each to manage among Docker containers and hosting
servers.
Automated detection
Automated Tracking
Monitor/Job management
GUI window

Docker container configuration management
Docker
container list
Docker hosts list
Map
Docker hosts
and
Docker
containers

Docker configuration management tag recognition
Operation and management configuration follows dynamic container
deployments.
Automated service group detection with Docker label such as Web or DB,
resulted scope and grouping automation.
Docker
Label: web container
Label: app container
Hinemos scope for Web servers
Hinemos Scope for App servers
Automated
grouping
Automated
grouping
Web server
monitoring
App server
Monitoring
Hands free
management

(3)Docker monitoring features
Monitoring metrics for Docker host and guest.
Container A start
Operation events
Manage container events
Container A
Container C
CPU#1
Load balances
Container B start
Container C Start
Container A Delete
Container D start
Detect
unwanted
Operations
CPU#2
Container B
Contianer A
Container C
Unbalanced
load

Docker Monitoring
• Monitoring for Docker Architecture
• Monitoring All Docker hosts
• Monitoring precisely for containers.
Docker Host Docker Container
Status
Resources
Events
Status
resources
Docker monitoring by Hinemos

Docker Monitoring: docker hosts
Docker Host status … Status=0(OK)/1(NG)
Docker Host CPU load … DockerホストのOS全体のCPU使用状況
Container numbers … Count per statuses (running/paused/exited/restarting)
Disk Usage by Docker … Usage per status

Docker Monitoring: precise resource monitoring
Monitor CPU load per
container
Monitor Core loads per
container

White Paper for Docker with Hinemos
Hinemos white paper for docker usage.
Also share
sample scripts

Resulted Architecture
Database Provisioning
Automation
Puppet
Ansible
Monitor/Job
Operation
Container
Image
management
(docker-registry)
Container Host
App
Brid
ge
NAT
consu
l
Hinemos agent
Container Host
App
Brid
ge
NAT
consu
l
Hinemos agent
Container Host
App
Brid
ge
NAT
consu
l
Hinemos agent
Container Host
App
Brid
ge
NAT
consu
l
Hinemos agent
registory
NAT
Application
Monitoring
search400
search001
…
Hinemos
manager

Distributed application usecase on docker

More Related Content

What's hot (20)

Similar to Distributed application usecase on docker (20)

More from Hiroshi Miura (20)

Recently uploaded (20)

Distributed application usecase on docker