Moving Data Between Exadata and Hadoop

www.enkitec.com++ 1+++
Moving'Data'Between'Oracle'Exadata'and'Hadoop.'
Fast.+
Tanel'Põder'
Enkitec'
+
h.p://www.enkitec.com+
h.p://blog.tanelpoder.com+

Intro:+About+me+
•  Tanel+Põder+
•  Former+Oracle+Database+Performance+geek+
•  Present+Exadata+Performance+geek+
•  Future+Hadoop+Perfomance+geek+
•  My+Exadata+experience+
•  2009+...+2013+
•  Exadata+V1+…+X3+
•  MulOPrack+Exadatas+
•  MixedPrack+Exadatas+
•  My+Hadoop+Experience+
•  Ask+again+next+year+;P)+
+
Expert'Oracle'Exadata'
book+
(with+Kerry+Osborne+and+
Randy+Johnson+of+Enkitec)+

About+Enkitec+
•  Enkitec+
•  North+America+
•  EMEA+
+
•  100++staﬀ+
•  In+US,+Europe+
•  Consultants+with++
Oracle+experience++
of+15++years+on+average+
•  What+makes+us+so+awesome+
•  200+'Exadata'implementaBons'to'date'
+
•  Enkitec+ExaPLab++
•  We+have+3+Exadatas+(V2,+X2P2,+X3P2)+
•  FullPRack+Big+Data+Appliance+
•  ExalyOcs+
•  ODA+
Everything'Exa'
'
Planning/PoC+
ImplementaOon+
ConsolidaOon+
MigraOon+
Backup/Recovery+
Patching+
TroubleshooOng+
Performance+
Capacity+
Training+

Our+exaPlab+environment+
•  Exadata+V2+(quarter+rack)+
•  Exadata+X2P2+(quarter+rack)+
•  Exadata'X3G2'(quarter'rack)'
•  Big'Data'Appliance'(full'rack)'
•  ExalyOcs,+ODA,+etc+
IB+

Disclaimers++
•  The+numbers+shown+here+are+not+from+"real"+benchmarks+
•  The+actual+data+loading+speeds+vary+greatly+when+using+real+data+
•  (column+count,+datatypes+etc+etc)+
•  This+is+not+a+"how+to+conﬁgure+hadoop+tools"+session+
•  ...it's+all+about+performance+

(Too)+Many+Data+Loading+OpOons+
•  Pull+Hadoop+data+into+Oracle+
•  Oracle'SQL'Connector'for'HDFS'
•  Oracle+Heterogenous+Services+++Hive/Impala+ODBC+
•  FusePmounted+HDFS+++external+table+load+
•  Push+Hadoop+data+into+Oracle+
•  Sqoop+
•  Oracle+Loader+for+Hadoop+
•  Pull+Oracle+data+into+Hadoop+
•  Sqoop+
•  Tom+Kyte's+ﬂat+unloader+(to+Hadoop+local+ﬁlesystem+++copy+to+HDFS)+

Oracle+SQL+Connector+for+HDFS+
CREATE TABLE "TANEL"."TERASORT_1T_100"
( "TOKEN_TYPE" VARCHAR2(4000),
"DATE_MONTH" VARCHAR2(4000),
"TOKEN_COUNT" VARCHAR2(4000),
"TOKEN_VALUE" VARCHAR2(4000)
)
ORGANIZATION EXTERNAL
( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "EXT_HDFS_TEST_DIR"
ACCESS PARAMETERS
( RECORDS DELIMITED BY 0X'0A'
PREPROCESSOR "OSCH_BIN_PATH":'hdfs_stream'
FIELDS TERMINATED BY 0X'3058273927'
( "TOKEN_TYPE" CHAR(4000),
"DATE_MONTH" CHAR(4000),
"TOKEN_COUNT" CHAR(4000),
"TOKEN_VALUE" CHAR(4000)
)
)
LOCATION
( 'osch-tanel-00000',
'osch-tanel-00001',
'osch-tanel-00002',
'osch-tanel-00003'
)
) ...
Visible+to+Oracle+as+an+
External+Table.+
Parallelizable.+Insert+select,+
CTAS+
The+PREPROCESSOR+
program+hdfs_stream+is+a+
java+program+capable+of+
reading/streaming+ﬁles+from+
HDFS+
The+Oracle+SQL+Connector+
Data+"locaOon+pointer"+ﬁles+
to+1'TB+of+data+

OSCH+data+locaOon+files+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<locationFile>
<header>
<version>1.0</version>
<fileName>osch-20130708020324-4644-1</fileName>
<createDate>2013-07-08T14:03:24</createDate>
<publishDate>2013-07-08T02:03:24</publishDate>
<productName>Oracle SQL Connector for HDFS Release 2.1.0 - Production</productName>
<productVersion>2.1.0</productVersion>
</header>
<uri_list>
<uri_list_item size="10000000000" compressionCodec="">
hdfs://enkbda-ns/user/acolvin/terasort/part-00000
</uri_list_item>
</uri_list_item>
</uri_list_item>
</uri_list_item>
</uri_list_item>
...
Each+"locaOon+pointer"+file+
the+external+table+loader+
uses+points+to+one+or+more+
actual+HDFS+files+
+
(this+config+file+is+edited+for+
fomaong+purposes)+

TesOng+Oracle+SQL+Connector+for+HDFS+
•  CREATE+TABLE+target+AS++
SELECT+/*++PARALLEL+*/+*+FROM++terasort_1t;+
Only+75+MB+per+
second?+

Where+is+your+bo.leneck?+
Hadoop+Cluster+
HDFS+
MR+
job+
MR+
job+
MapReduce+
(+CPU+)+
Oracle+Database+
Storage+
MR+
job+
MR+
job+
PX+Slaves+
(+CPU+)+
I/O+
O/I+
Network+
+
+
"ComputaOon"+
Decompression+
Text+file+parsing+
Datatype+conversion+
Text+file+parsing?+
Datatype+conversion?+
HCC+compression?+
DB+Waits+
ContenBon?'
+
Network+bandwidth+/+
throughput+/+
configuraOon++
The'only'way'to'
know'is'to'measure!'

TesOng+Oracle+SQL+Connector+for+HDFS+

Unbalanced+Parallel+Slave+acOvity?+

Increase+Max+Allowed+External+Table+Parallelism+
CREATE TABLE terasort_1t_100 (
...
ORGANIZATION EXTERNAL
( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "EXT_HDFS_TEST_DIR"
...
PREPROCESSOR "OSCH_BIN_PATH":'hdfs_stream'
...
LOCATION
(
'osch-tanel-00000'
, 'osch-tanel-00001'
...
)
...
SoluOon:+Create+more+
"locaOon+pointer"+files.++
100+"locaOon+pointer+files",+
each+poinOng+to+a+single+
HDFS+file+(in+my+test)+
This+allows+up#to+100+slaves+
in+parallel,+accessing+one+
HDFS+stream+each.+

More+"finePgrained"+OSCH+data+locaOon+files+
$ cat osch-tanel-00099

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<locationFile>
<header>
<version>1.0</version>
<fileName>osch-tanel-00099</fileName>
<createDate>2013-07-08T14:03:24</createDate>
<publishDate>2013-07-08T02:03:24</publishDate>
<productName>Oracle SQL Connector for HDFS Release 2.1.0 - Production</productName>
<productVersion>2.1.0</productVersion>
</header>
<uri_list>
</uri_list_item>
</uri_list>
</locationFile>

$ ls -l osch-tanel*
-rwxr-xr-x 1 nobody users 598 Sep 24 12:07 osch-tanel-00000
...

100+files,+allowing+up'to+100+
HDFS+streams+in+parallel.+
+
With+less+PX+slaves,+each+
slave+can+access+mulOple+
files+sequenOally.++

BDA+P>+Exadata+X3P2+(16core/32thread)+1TB+data+load:+
500P600+MB/s+load+by+single+
DB+node+(1P2+TB+hour)++

BDA+P>+Exadata+X3P2+(16core/32thread)+1TB+data+load:+
Skewed/Unbalanced+parallel+
execuOon:+4+slaves+work+for+
longer+when+others+are+
done+(4+x+32+++4+=+100+ﬁles)+

Hadoop+Cluster+CPUs+are+idle?!+

Drilling+deeper+into+the+CPU+usage+
SQL> @ostackprof 788 0.1 100

Below is the stack prefix common to all samples:
------------------------------------------------------------------------
Frame->function()
------------------------------------------------------------------------
# 49 ->main()
.... some lines snipped .....
# 11 ->pextproc()
# 10 ->spefmccallstd()
# 9 ->spefcpfa()
# 8 ->qxxqFetch()
# 7 ->kpxsFetch()
# 6 ->kpxsFetchField()
# 5 ->kpxsFetchDriver()
.... some lines snipped .....
# -#--------------------------------------------------------------------
# - Num.Samples -> in call stack()
# ----------------------------------------------------------------------
35 ->kudmxfe()->kudmdtp()->lxoSchPat()
25 ->kudmxfe()->kudmdtp()->lxmfwdx()
23 ->kudmxfe()->kudmdtp()->
4 ->kpxsDoConvert()->OCIDirPathColArrayToStream()->kpudpcs_colArrayToStream()-
>kpudpcsf_intColArrayToStream()
3 ->kudmxfe()->lxmfwdx()
3 ->kudmxfe()->kudmrn()->kudmrt()
2 ->qerxtCBFetch()->qerxtProcessRows()->qeaeCn1Serial()
2 ->qerxtCBFetch()->qerxtProcessRows()->klxprParseRow()
1 ->OCIDirPathColArrayReset()
83%+of+Ome+spent+in+
datatype+conversion+(kudm)+
++
60%+in+lx*+funcOons+–+string/
datatype+processing++

Datatype'Conversion'is'CPU'hungry!!!'
You+can+oﬄoad+the+
"preprocessing+and+datatype+
conversion"+to+the+Hadoop+
cluster+CPUs+with+the+Oracle'
Loader'for'Hadoop!'

Oracle+Loader+for+Hadoop+
Hadoop+Cluster+
HDFS+
MR+
job+
MR+
job+
MapReduce+
(+CPU+)+
Oracle+Database+
Storage+
MR+
job+
MR+
job+DB+Process+
I/O+
O/I+
With+OCI/DataPump+
it's+possible+to+
convert+data+to+
Oracle+naOve+format+
No+datatype+
conversion+needed+
HCC+compression?+
DB+Waits+
ContenBon?'
+
Array+insert+(JDBC)+
Direct+Path+Load+(OCI)+
Create+DataPump+ﬁle+
(load+via+ext+table)+
Already'preG
converted'data'is'
sent'to'Oracle'

•  Source:(High(Performance(Connectors(for(Load(and(Access(of(Data(from(
Hadoop(to(Oracle(Database((
•  June+2012+
•  h.p://www.oracle.com/technetwork/bdc/hadoopPloader/connectorsPhdfsP
wpP1674035.pdf+
Based+on+earlier+tests,+
these+numbers+are+
plausible.+(although+your+
mileage+will+vary+
depending+on+the+data+
you+convert+and+load)+

Oracle+Loader+for+Hadoop+
•  Can+preprocess+and+convert+datatypes+to+Oracle+"naOve"+
format+using+Hadoop+cluster's+CPU+cycles+
•  DataPump+format+
•  OCI+Direct+Path+load+format+
•  Each+Reducer+in+Hadoop+connects+to+Oracle+DB+with+a+
separate+session+(OCI/JDBC)+
•  So+OCI+direct+path+loads+must+be+done+into+parOOoned+tables!+
•  Otherwise+you'll+get+TM+enqueue+contenOon+
•  Oracle+Loader+takes+care+of+the+distribuOon+
•  As+long+as+you+have+enough+reducers+conﬁgured+

References+
OTN+Big+Data+Connectors+page+
•  h.p://www.oracle.com/technetwork/bdc/bigPdataPconnectors/
overview/index.html+
Oracle+Big+Data+Connectors+User's+Guide+
•  h.p://docs.oracle.com/cd/E41604_01/doc.22/e41238/toc.htm+
•  Tools+
•  dstat+
•  h.p://dag.wieers.com/homePmade/dstat/+
•  SwingBench+CPU+Monitor+
•  h.p://www.dominicgiles.com/cpumonitor.html+
+
+

Thanks!!!+
•  QuesOons?+
•  Ask+now+:)+
•  Or+Contact+
•  tanel@tanelpoder.com+
•  h.p://blog.tanelpoder.com+
•  @tanelpoder+
+
•  h.p://www.enkitec.com+
•  We+rock!+;P)+

Moving Data Between Exadata and Hadoop

More Related Content

What's hot (19)

Similar to Moving Data Between Exadata and Hadoop (20)

More from Enkitec (20)

Recently uploaded (20)

Moving Data Between Exadata and Hadoop