Hadoop eco story 이해

BVYdde ?Xd Midgn
gZZehl
.,-2* ,/
-

-* BVYdde
BVYdde :
) Cc -553( >dj =jii c ( FjXZcZ ~ ( c .,,,( deZc hdjgXZ
) Cc .,,-( eVX Z HjiX ~ ( QZW =gVlaZg
) - XdgZ( -A< L G( L C> aZkZa - dc 4 YY idiVa -N<
) CcYZm c gViZh 8 -,, eV Zh + hZX
) a b iZY i Z idiVa cjbWZg d eV Zh id -,, b aa dc
) QZW cYZm c w
) 0 w x
) y Y h y
CcYZm c *
N Z hidgn d BVYdde .

B> M :
eVX Z HjiX 0~
hX ZbVaZhh( YjgVWaZ( VcYa c V ajgZ( VjidbVi Vaan gZWVaVcXZY
Cc .,,/( Add aZ ejWa h ZY i Z Add aZ aZ MnhiZb eVeZg
Cc .,,0( YZkZadeZY HjiX > hig WjiZY aZ MnhiZb
) >jgVWaZ 6 20G WadX (
) BVcYa c V ajgZ 6 z HdYZ
) jidbVi XVaan gZWVaVcXZ 6 HVbZ HdYZ
/

B> M :
0iie6++lll*ha YZh VgZ*cZi+bVYk gjh+ Y h).443-012

GVeLZYjXZ :
H> M :(
) x (
) xx (
) *
! Cc .,,0( uGVeLZYjXZ6 M bea ZY >ViV JgdXZhh c dc
FVg Z =ajhiZghv
2
GVe
HdYZ
LZYjXZ
GVe

GVeLZYjXZ :
3
-* ldgYXdjci GVeLZYjXZ *
.* ( Mea i( ( ( ( …
/* /~ Mea i *
0* bVe … 7 (-9 *
1* … Zn
*
2* gZYjXZ … kVajZ hjb
cVa kVajZ *
3*
<ZVg(. =Vg(/ >ZZg(. L kZg(.
cVa 7 Zn(kVajZ9 w

4
RackN
NodeManager
NodeManager
NodeManager
Rack2
NodeManager
NodeManager
NodeManager
Rack1
NodeManager
NodeManager
NodeManager
C2.1
C1.4
AM2
C2.2 C2.3
AM1
C1.3
C1.2
C1.1
Hadoop Client 1
Hadoop Client 2
create app2
submit app1
submit app2
create app1
ASM Scheduler
queues
ASM Containers
NM ASM
Scheduler Resources
.......negotiates.......
.......reports to.......
.......partitions.......
ResourceManager
status report
GVeLZYjXZ :

BVYdde !
.,,2 BVYdde !
) B> M GVeLZYjXZ BVYdde =dgZ a W
) .,,2 S BII
.,,3 Nl iiZg( VXZWdd ( F c ZYCc
.,,4 MjW Jgd_ZXiw
) BWVhZ( Tdd ZZeZg( J SV dd ( B kZ VXZWdd q
) =adjYZgV Add aZ( VXZWdd ( SV dd ~
.,,5 bVodcZ GVeLZYjXZ
.,-- Bdgidcldg h SV dd ~
5

BVYdde** q
GVeLZYjXZw !!!!
) B> M
) =ajhiZg (
) DdW ( egdXZhh c
) =a Zci JC q!
<ViX :
) KjZgn( AgVe JgdXZhh c
** q:
-,

BVYdde** q w !
w :
) **
) ** **
B> M GVeLZYjXZ :
.,-. GVeLZYjXZ P. S LH
--

BVYdde .*, :
=ajhiZg …
GVeLZYjXZ w
BVYdde
S LH
-.

BVYdde MQw :
-/
N Z BVYdde ?XdhnhiZb NVWaZ
.,-2
-2, ~ MQ
iie6++ VYddeZXdhnhiZbiVWaZ* i jW* d+

BVYdde ?Xd r
-0
B> M
S LH
y
HdMKF
MKF dc BVYdde
x
BVYdde)gZaViZY egd_ZXih Vi eVX Z
HZlMKF
x

BVYdde ?Xd ( MKF dc BVYdde
-1
w +
!!
:
B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
J ( B kZ( Mi c Zg NZo ( NV_d(
CbeVaV( JgZhid( BVl
Add aZ >gZbZa
.,--
>ViV

BVYdde ?Xd ( MKF dc BVYdde
-2
• J .,,4 6 MXgei GVeLZYjXZ
• BkZ .,,4 6 MKF GVeLZYjXZ
• >g aa .,-- 6 B> M B<VhZ(Gdc d><(M/
GVeL
• CbeVaV .,-. 6 y ( ( =adjYZgV
• JgZhid .,-/ 6 _dcw VXZWdd
• B kZ .*, .,-2 6 B kZ NZo =C> ( ~
• NV_d .,-0 6 >Q dc BVYdde( MKF
• MeVg MKF .,-0 6 y
• B QK .,-/ 6 GJJ (
sMKF t MKF)Ic)BVYdde : MKF -,w

: y q
-3
B> M
GVeLZYjXZ
J MXg ei
B kZ MKF
B kZ .*, B kZ iZo
>g aa( JgZhid
CbeVaV y
NZo
MeVg kh NZo
NV_d
BVlf
~
MeVg MKF

q
-4
y …
y
NV_d( B kZ
w:
MKF
>g aa( CbeVaV( JgZhid( B QK(
B kZ dc NZo
!
MeVg L>>
MeVg MKF L>>( B kZ
y

q
-5
MnhiZb HVbZ
Vjai
NdaZgVcXZ
>ncVb X
MX ZYja c
Fd N bZ
KjZgn
Fdl
FViZcXn
eVX Z NV_d I I I I
CbeVaV R R R I
Mig c Zg
B kZ
I I I I
JgZhid R R R I
MKF)dc)BVYdde :

MKF dc BVYdde
.,
MKF)Ic)BVYdde y *
78 B> M & WVcYl Yi
-, ( 4~( M N 3.,, geb
-, & 4~ & -,,G<+hZX 8 4A<+hZX
-*1N< B kZ( NV_d( CbeVaV
*
hZaZXi Xdjci & gdb NUN?MN
VXidg " "( "
y"( " " * w w
w w VXidg * B kZ -
NV_d( CbeVaV -*1 p / * -,, ( .,, w
* Q Vi YdZh u-,, i bZh VhiZg i Vc B kZv VXijVaan bZVc:

MKF)Ic)BVYdde VXidg
.-
• BVYdde 6 jaa hXVc
• y 6 y gdje Wn( dgYZg Wn
• … 6 dgYZg Wn kZ
gZYjXZ
w *
• 6 L= aZ ( IL= aZ( eVgfjZi
• 6 IL=
*
• 6 (
• y 6

BVYdde :
..*** ”
IL= JVgfjZi
Bdgidcldg h
(
DVkV (
L= aZ
=adjYZgV
( (
**
(
McVeen
? X Zci >ViV MidgV Z dg cVani Xh l i eVX Z JVgfjZi .*,

K “ **
./
K JVgfjZi WadX h oZw Y h WadX h oZ w : ?LL 6 JVgfjZi aZ
h djaY cdi WZ hea i cid bjai eaZ Y h)WadX h
è B kZ eVgfjZi ( beVaV
è JVgfjZi Y h WadX h oZ
iie6++hiVX dkZg adl*Xdb+fjZhi dch+/,404331+hZi)eVgfjZi)hcVeen)djieji) aZ)h oZ) h) kZ

BVYdde ?Xd ( HdMKF :
.0
HdMKF y ~
HdMKF B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
BVYdde ><GM q
Add aZ < NVWaZ
.,-.
B> M >< :
BWVhZ( =VhVcYgV( Gdc d><

HdMKF w :
.1Gdc d><( =VhhVcYgV( B<VhZ y
• =VhcVYgV .,,4 6 VXZWdd
• B<VhZ .,,2 6 ><
• XXjbjad .,-- 6 B<VhZ ( ”
• LZYh 6 ><
• ?aVhiXMZVgX .,-, 6 FjXZcZ
• Gdc d>< 6 -, Zc( w ( Ddc w
• >ncVbd>< 6 QM ( ( MM>
y(
) =VX Z LZY h ><
) ZaVhi XhZVgX

HIMKF!! = J :
.2HdMKF y !
=dch hiZcXn 6 xx w 8 w y
*
kV aVW a in 6 w 8 w
JVgi i dc NdaZgVcXZ 6 8 HQ
y
HdMKF J w =
=
L><GM
=J
w (
B<VhZ
J
>ncVbd( =VhVcYgV( =djX ><
= J = J NZdgZb HdMKF

HIMKF :
B> M
w
) y w ) w
HdMKF y ~
.3

HdMKF BWVhZ
.4
iie6++lll*ijidg Vahed ci*Xdb+ WVhZ+ WVhZUijidg Va*eY
BWVhZ B> M HdMKF ><

HdMKF
/,
HdMKF w è u v q :
>< w ( F hi
HdMKF y ~
KjZgn

HdMKF :
/-
…
-(
.* 6 8 - 6 -
/* 6 8 - 6 H
0* y
L><GM
… !
HdMKF

HdMKF :
/.
EZn)PVajZ *
PVajZ 0w * ><
PVajZw Mig c (
F hi( GVe
LZY h( >ncVbd><(
L V
EZn)PVajZ
PVajZ ~
=dajbc
BWVhZ( =VhVcYgV(
>gj Y( XXjbjad
=dajbc Vb an
PVajZ mba( Dhdc
Gdc d><( =djX ><(
?aVhi XhZVgX
>dXjbZci

HdMKF
//
iie6++c.b*Xd* g+c.b.,-,+YdlcadVY+=djX <VhZP-U0*eY
• Dd c Dd c w ( HdMKF
… >< _d c
• HdMKF Dd c w w

HdMKF >dXjbZci
/1
=dajbc
Vban
>dXjbZci

K =dajbc >< :
/2>< w: Xdajbc dg ZciZY YWbh
=dgZ
w *
)
)

HdMKF MKF dc BVYdde** w :
/3
HdMKF Hdi Ican MKF … ><GM
><
< M?
MKF
HdMKF
L><GM
MKF dc BVYdde
…
= J HdMKF y !
(

HdMKF MKF dc BVYdde** !
/4
B kZ MKF FVnZg :
B> M
S LH >ViV IM
B< M? B kZ
MKF =a Zci
CchZgi + OeYViZ >ZZe MKF cVanhh
B<VhZ VcY B kZ r <ZiiZg Nd Zi Zg
ü B< M?
>ViV
ü
ü BkZ
ü
KjZgn

HdMKF
w
:
HdMKF ee *!
è ‘ >< ee w *
ee ~ w * ~
P Zl
P Zl
à w
HdMKF
:
J
w =J
/5

HIMKF!! = J : < M? :
0,
<Vh XVaan kV aVWaZ 6 kV aVWaZ
Md i)hiViZ 6 w >ViVw Zme gZ
?kZcijVaan Xdch hiZcXn 6 y >ViV
= J
< M?
HdMKF y !
=dch hiZcXn 6 xx w 8 w
y *
kV aVW a in 6 w 8 w
*
JVgi i dc NdaZgVcXZ 6 8
*

HIMKF!! = J : < M? :
0-
=C> * (
. … w *
< M? ~ * =C>w ~
( < M? *
=C>( < M? = J
= J /~ .~ *
J ( *
‘( ‘ Jw w =
* = : =dch hiZcXn
?kZcijVaan =dch hiZci ~ *
HdMKF y ~ r ?kZcijVaan =dch hiZci

HZl MKF:
0.HZlMKF :
B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
HdMKF
HZlMKF
=C>
Hdc)adX c
h VgZY)cdi c
HdMKF VcY
L><GM …
:
Pdai><( Hjd><( bZbMKF( =ajhig m

HZl MKF :
0/
HdMKF L><GM
MKF
=C>
< o HZlMKF ><GM
(
HZlMKF y
HZlMKF LZXdgY FdX c
ad c
<j Zg edda

HZl MKF :
00
- MKF
. igVchVXi dch =C> WZ c( gdaaWVX ( Xdbb i
/ Hdc)adX c XdcXjggZcXn Xdcigda
NgVchVXi dc FdX c
Hdc)adX c w *
& ( HZlMKF w
0 B eZg)cdYZ eZg dgbVcXZ
x ><GM *
1 JVgVaaZa( h VgZY)cdi c VgX iZXijgZ
(
w x *

HZlMKF
01
GZbdgn
bnMKF
ü Pdai>< AHO ( GZbMKF
ü MXVaZ>< GnMKF hXVaVW a in (
ü Nd j>< AHO GVg V>< hXVaaVW a in
ü =ajhig m> Y hig WjiZY eZZg)id)eZZg MKF YViVWVhZ
ü Hjd>< Y hig WjiZY YViVWVhZ bVcV ZbZci
s-0 2 HZlMKF
=ajhig m( bZbMKF( Hjd><( Mea XZ GVX cZ( NgVch FVii XZ( Pdai><
-2 HdMKF( HZlMKF >ViVWVhZh Nd QViX

HZlMKF( HdMKF( L><GM :
02
ü L><GM dg cZlMKF
ü HdMKF dg HZlMKF
ü L><GM
ü HdMKF
ü < o( Cc gV

HZlMKF :
04
FZVYZg’ IgVXaZ(
G Xgdhd i( M J( C<G H X Z JaVnZg w
…
uIcZ h oZ YdZh cdi i Vaav

05
B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
HdMKF
(
BVYdde
(
q
BVYdde
:
ajbZ( H ( ajZciY( ad hiVh
HZlMKF

r w :
1,
• MXgWZ .,,4 6 VXZWdd (
• = j lV .,,4 6 SV dd(
• ajbZ .,-. 6 ( y ?NF
• ajZciY 6 ajbZ eaj c ( ? E w
• Fd hiVh 6 * * w
• H .,-0 6 ‘ HM
• MKIIJ 6 L><GM BVYdde
• EV V 6 Fc ZYCc( GZhhV Z KjZjZ >h
< o Fd ( (
) y EV Vw *
? E 6 ?aVhi XhZVgX ajZciY E WVcV Fd gZ Vidg
FIA
L><
GMA
KjZjZ

r :
1-
w
MdjgXZ )9 = VccZa )9 Mc
) MdjgXZ 6
) = VccZa 6
) Mc 6
Jaj c
Jaj c ~
w
) 6 ><( JdgdidXda( MHM( B> M
) 6 ><( KjZjZ(

r :
1.
:
JgdidXda
DMIH NZmi RGF
< cVgn DMIH y w(
hoZ w( NZmi JVghc
YViV dgbVi
N g i( kgd( JgdidWj (

r :
1/
egdidWj w ( i g iw *
PLI
)
) (
i g i
) .,,3( VXZWdd
) ( w *
egdidWj
) .,,4( Add aZ
) w ( w
N g i kh JgdidXda <j Zgh kh kgd ) < VhZY =dbeVg hdc

y + MigZVb c
10
B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
HdMKF
higZVb c
:
GZbgdn
?kZci
GL …
) C+I HQ( > h (
MeVg ( Midgb( hoboV
HZlMKF
y

y r w :
11
• M0 .,-- 6 SV dd( higZVbc .,-/
• Midgb .,-/ 6 NliiZg( higZVbc (
• MVboV .,-0 6 Fc ZYCc( higZVbc ( >h
• JjahVg .,-1 6 ZWVn( higZVbc =?J( >gjY
• MeVg .,-0 6 >ViV<gX h( L>> + ( GF+MKF
• a c .,-0 6 YViV gihVch( MeVg ( egdXZhh
MeVg a W
y MigZVb c < >ViV cZmi hiZe dg eVX Z a c
a c

y w:
12
GL
NZo
MeVg
a c
>ViV
>ViV
MVboVMidgb
JjahVg
B> M( aZ HdMKF
…
MHM( =Vaa(
> A
BVYdde
> A
L>>

13NZo
B> M *
GVe LZXYjXZ *
GVeLZYjXZ kh NZo 6 ~ :

14
6 GZbdgn ( > A
<ViX MigZVb c w G Xgd <ViX
GVeLZYjXZ kh MeVg 6 ~ :

15
MigZVb c < >ViV6 Midgb( MeVg VcY MVboV
6 GZbdgn ( > A
<ViX MigZVb c w G Xgd <ViX
eVX Z Midgb

Qdg adl( Gdc idg c
20
BVYdde
w OC …+
MQ !!!!
:
bWVg ( =adjYZgV GVcV Zg
Ido Z( o VWVc( Fj
BVYdde
>ViV
B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
HdMKF
HZlMKF
y

Qdg adl
21
OC
o VWVc
F c ZYCc
BVYdde
o VWVc
Ido Z
RGF
OC
Fj
Jni dc
=dYZ
Fj
Jni dc ~
Qdg adl ?c cZ =dbeVg hdc

22
BVYdde =ajhiZg
=adjYZgV GVcV Zg
=adjYZgV
bWVg
BdgidcQIg h
aVb c d
‘ =adjY cZ
< >ViV

Hadoop eco story 이해

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to Hadoop eco story 이해 (20)

More from SANG WON PARK (17)

Recently uploaded (20)

Hadoop eco story 이해