SlideShare a Scribd company logo

LCA13: Hadoop DFS Performance

Linaro, profile picture
Linaro

The document discusses optimizations made to improve the performance of Hadoop Distributed File System (HDFS) on ARM platforms. It details the testing methodology used and various optimizations tried, including using NEON to optimize the CRC32c checksum algorithm, replacing the PureJavaCrc32C class with a native implementation, and using direct buffers for data writes. Benchmark results show a reduction in CPU usage for both read and write paths after applying these optimizations. Further optimizations to disk I/O and tracking of Java virtual machines are suggested.

Technology
1 of 26
Downloaded 16 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
ASIA 2013 (LCA13) LEG – Hadoop DFS Performance Steve Capper <steve.capper@linaro.org>
ASIA 2013 (LCA13) www.linaro.org Hadoop Performance on ARM I concentrated on reducing the time taken for CPU bound tasks (the latency). My work, so far, has focussed on the underlying cluster filesystem, HDFS, as this underpins a lot of Hadoop workloads. The latest release-tagged Hadoop at the time of experimentation was 2.0.2-alpha. So this was the version used. (2.0.3-alpha has just come out). Hadoop installations consist of rather a lot of moving parts. So please let me know if I should be concentrating on something else! :-)
ASIA 2013 (LCA13) www.linaro.org Hadoop Distributed Filesystem – Overview HDFS is the default Hadoop distributed filesystem. Consists of multiple “nodes”: Namenode – holds the metadata for the filesystem and keeps track of the datanodes. Only one namenode is active at a time per namespace. Thus there is a high memory requirement for a namenode. Datanodes – store the file's blocks. The default block size is 64MB. (Optional) Passive namenodes – To maintain an High Availability configuration, one can set up shared storage (via NFS or by journalnodes) and have namenodes on standby to failover. Filesystem blocks are replicated between datanodes. Datanodes can be “rack aware”; data will be distributed between racks. Nodes can run on the same machine or on different machines.
ASIA 2013 (LCA13) www.linaro.org Data Integrity in HDFS Hadoop mitigates against hardware failure in the following ways: Metadata can be saved to multiple filesystems and regular snapshots are usually taken to allow for disaster recovery. On some HA configurations, multiple journalnodes maintain a quorum of the metadata. Data blocks are replicated across multiple datanodes (preferably to different racks). Data blocks are regularly transmitted: All data streams are checksummed. The default checksum algorithm is CRC32c. The default number of bytes per checksum is 512 bytes.
ASIA 2013 (LCA13) www.linaro.org Test Configuration So far I have been micro-benchmarking single machine workloads (1 name node, 1 data node, 1 workload). I have been working on an A9 platform. TestDFSIO -read and -write have been tested with a 10GB filesize. Soft-float Oracle JDK 1.7.0_10 was used. Performance was measured using Linux perf -a, Java samples were taken with the inbuilt profiler: -Xrunhprof:cpu=samples,depth=10
ASIA 2013 (LCA13) www.linaro.org Plain TestDFSIO write tests – Samples CPU SAMPLES BEGIN (total = 164879) rank self accum count trace method 1 24.85% 24.85% 40970 301219 sun.nio.ch.EPollArrayWrapper.epollWait 2 24.79% 49.64% 40870 301362 sun.nio.ch.EPollArrayWrapper.epollWait 3 24.79% 74.42% 40866 301363 sun.nio.ch.EPollArrayWrapper.epollWait 4 24.78% 99.20% 40859 301358 sun.nio.ch.EPollArrayWrapper.epollWait 5 0.07% 99.27% 118 300073 java.lang.ClassLoader.defineClass1 6 0.03% 99.31% 57 300020 java.util.zip.ZipFile.open 7 0.03% 99.34% 55 300179 java.util.zip.ZipFile.getEntry 8 0.03% 99.38% 54 300062 java.util.zip.ZipFile.read 9 0.02% 99.39% 31 300076 java.util.zip.Inflater.inflateBytes 10 0.01% 99.41% 23 301165 sun.nio.ch.FileDispatcherImpl.force0 + 33.80% java perf-13641.map [.] 0xb4258538 + 19.98% java perf-13534.map [.] 0xb403e980 + 5.37% java libc-2.15.so [.] memcpy + 3.02% java [kernel.kallsyms] [k] __copy_from_user + 2.60% java [kernel.kallsyms] [k] __copy_to_user_std + 1.86% java [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore + 1.01% swapper [kernel.kallsyms] [k] finish_task_switch + 0.66% swapper [kernel.kallsyms] [k] tick_nohz_idle_enter + 0.55% java [kernel.kallsyms] [k] get_page_from_freelist + 0.44% java [kernel.kallsyms] [k] free_hot_cold_page + 0.42% swapper [kernel.kallsyms] [k] default_idle Namenode Java samples: Linux perf results:
ASIA 2013 (LCA13) www.linaro.org Plain TestDFSIO read & write CPU usage Plain TestDFSIO Write Plain TestDFSIO Read Other Java memcpy crc32c_sb8
ASIA 2013 (LCA13) www.linaro.org Some analysis of plain TestDFSIO The namenode doesn't do much when it's in charge of 1 datanode. (this will change when things scale up). When writing data: Most of the Java time is spent in PureJavaCrc32C.update Most of the CPU time is spent in Java land. When reading data: Most of the Java time is spent in NativeCrc32.nativeVerifyChunkedSums. Most of the CPU time is in the native function: crc32c_sb8 For both reading and writing data: ~10% of the CPU time is spent copying memory.
ASIA 2013 (LCA13) www.linaro.org Optimising the read and write paths Trevor Robinson has proposed a patch to improve the write path: HDFS-3529 “Use direct buffers for data in write path” I have been working on speeding up the computation of CRC32c checksums, by using NEON. Also, I've performed some very preliminary experiments on replacing PureJavaCrc32C with a JNI class that references the NEON CRC code.
ASIA 2013 (LCA13) www.linaro.org NEON optimisation of the CRC32c algorithm I worked on an algorithm Q4 last year: https://wiki.linaro.org/LEG/Engineering/CRC Given an input buffer, we perform polynomial multiplication and addition to give a slighter smaller buffer with the same CRC. This is referred to as “folding”. The algorithm reduced the buffer by 32 bytes at a time. The final buffer of 32 bytes was then processed by slice-by-8 as normal. I found that the vast majority of CRCs in Hadoop (in both NativeCrc32.nativeVerifyChunkedSums and PureJavaCrc32C.update) were computed for 512 byte buffers.
ASIA 2013 (LCA13) www.linaro.org A single 64 bit fold 32 64 64 M(x)32 A(x)B(x) 64 =A(x) * (x^64 mod P(x)) =B(x) * (x^96 mod P(x)) M'(x) + + 64 CRC(M'(x)) = CRC(M(x)) =
ASIA 2013 (LCA13) www.linaro.org NEON implementations With ARMv7 NEON we can only perform polynomial multiplication on 8 bit lanes: We need to be able to multiply at least 32 bits, thus multiple vmull.p8's were chained together to achieve this). There are 16 vmull.p8's per fold, so a little register pressure! A gcc intrinsic version has been coded up: This is considerably simpler to look at. Unfortunately, it runs a little slower than hand-optimised assembler. A test case has been sent to the Linaro Toolchain Working group (a couple of weeks ago) and this is being analysed by them.
ASIA 2013 (LCA13) www.linaro.org Replacing PureJavaCrc32C PureJavaCrc32C has two noteworthy methods: public void update(byte[] b, int off, int len) – called mostly for lengths of 512. public void update(int b) – seldom called. I created a new class implementing the Checksum interface that: Used the same implementation of: update(int b) as PureJavaCrc32C. Called straight into JNI for update(byte[] b, int off, int len). The class name NativeCrc32 was already taken by something else, so I chose a rather silly temporary name: HyperCrc32C
ASIA 2013 (LCA13) www.linaro.org Dealing with byte [] in JNI We are only reading from the byte[] array, and only for a very short time. Thus I pinned the buffer in memory with: GetPrimitiveArrayCritical Then subsequently released the buffer with ReleasePrimitiveArrayCritical(..., JNI_ABORT) This worked for me, but perhaps a better long term solution would be to change the backing data type to a ByteBuffer? We could also be clever and change the alignment of these? Rather than test every optimisation individually in this talk; I am going to put them all together.
ASIA 2013 (LCA13) www.linaro.org TestDFSIO read & write CPU usage Plain TestDFSIO Write New TestDFSIO Write Other Java memcpy crc Plain TestDFSIO Read New TestDFSIO Read
ASIA 2013 (LCA13) www.linaro.org Some Analysis of the New TestDFSIO runs The name node samples are unchanged. For the write path: Most of the time is now spent running native code rather than Java code. There is a noticeable reduction in Hadoop user CPU usage. For the read path: Most of the time is still spent running native code. There is again a reduction in Hadoop user CPU usage. For both the read and write paths there is a significant amount of CPU time spent copying memory around.
ASIA 2013 (LCA13) www.linaro.org Conclusions and Further Work The CPU usage required for TestDFSIO runs has been reduced for both read/write paths. This gives us more CPU cycles to run Hadoop jobs with! Hadoop is known to be very sensitive to underlying disk IO. To optimise HDFS IO, it would make sense to optimise disk/filesystem IO as much as possible and re-run these benchmarks. As Hadoop runs under Java: It makes sense to keep track of JVMs. A beta hard-float JVM has been released. The CPU usage for memcpy is making me uneasy!
ASIA 2013 (LCA13) www.linaro.org We should also consider our hardware... HDD SSD 0 0.5 1 1.5 2 2.5 TestDFSIO Write Relative performance MB/s HDD SSD 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 TestDFSIO Read Relative performance MB/s
ASIA 2013 (LCA13) www.linaro.org Plain TestDFSIO write tests – Some Samples CPU SAMPLES BEGIN (total = 81097) rank self accum count trace method 1 36.88% 36.88% 29906 300854 sun.nio.ch.EPollArrayWrapper.epollWait 2 32.95% 69.83% 26723 301107 sun.nio.ch.EPollArrayWrapper.epollWait 3 16.48% 86.31% 13363 301109 org.apache.hadoop.util.PureJavaCrc32C.update 4 2.74% 89.05% 2225 301113 sun.misc.Unsafe.copyMemory 5 1.67% 90.72% 1357 301115 sun.nio.ch.FileDispatcherImpl.write0 6 1.09% 91.81% 884 301128 org.apache.hadoop.hdfs.DFSOutputStream.writeChunk 7 0.70% 92.51% 564 301110 sun.nio.ch.FileDispatcherImpl.read0 8 0.42% 92.93% 342 301123 org.apache.hadoop.hdfs.DFSOutputStream.writeChunk 9 0.40% 93.33% 326 301202 org.apache.hadoop.hdfs.DFSOutputStream.waitAndQueueCurrentPacket 10 0.33% 93.66% 266 301124 java.lang.System.arraycopy CPU SAMPLES BEGIN (total = 227245) rank self accum count trace method 1 16.85% 16.85% 38293 301110 sun.nio.ch.EPollArrayWrapper.epollWait 2 16.82% 33.67% 38228 301313 sun.nio.ch.EPollArrayWrapper.epollWait 3 16.80% 50.48% 38184 301374 sun.nio.ch.EPollArrayWrapper.epollWait 4 16.80% 67.28% 38184 301375 sun.nio.ch.EPollArrayWrapper.epollWait 5 16.80% 84.08% 38169 301373 sun.nio.ch.ServerSocketChannelImpl.accept0 6 6.75% 90.83% 15342 301646 java.io.FileOutputStream.writeBytes 7 2.77% 93.60% 6294 301642 org.apache.hadoop.util.DataChecksum.verifyChunkedSums 8 2.49% 96.08% 5653 301664 java.io.FileOutputStream.writeBytes 9 1.14% 97.22% 2587 301640 sun.nio.ch.FileDispatcherImpl.read0 10 0.35% 97.57% 798 301691 sun.nio.ch.FileDispatcherImpl.write0 Workload Java samples: Datanode Java samples:
ASIA 2013 (LCA13) www.linaro.org Plain TestDFSIO read tests – Some Samples CPU SAMPLES BEGIN (total = 13668) rank self accum count trace method 1 36.82% 36.82% 5033 301083 org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums 2 31.37% 68.19% 4287 301080 sun.nio.ch.EPollArrayWrapper.epollWait 3 14.00% 82.18% 1913 301086 sun.nio.ch.FileDispatcherImpl.read0 4 4.82% 87.01% 659 301082 sun.misc.Unsafe.copyMemory 5 2.00% 89.00% 273 301118 sun.nio.ch.SocketChannelImpl.isConnected 6 1.87% 90.88% 256 301089 sun.nio.ch.FileDispatcherImpl.read0 7 0.76% 91.64% 104 300072 java.lang.ClassLoader.defineClass1 8 0.61% 92.25% 84 301105 sun.nio.ch.SocketChannelImpl.isConnected 9 0.51% 92.76% 70 300841 sun.nio.ch.EPollArrayWrapper.epollWait 10 0.49% 93.25% 67 301127 sun.misc.Unsafe.copyMemory CPU SAMPLES BEGIN (total = 118437) rank self accum count trace method 1 17.57% 17.57% 20810 301107 sun.nio.ch.EPollArrayWrapper.epollWait 2 17.51% 35.08% 20741 301304 sun.nio.ch.EPollArrayWrapper.epollWait 3 17.48% 52.56% 20697 301368 sun.nio.ch.EPollArrayWrapper.epollWait 4 17.48% 70.03% 20697 301369 sun.nio.ch.EPollArrayWrapper.epollWait 5 17.47% 87.51% 20695 301367 sun.nio.ch.ServerSocketChannelImpl.accept0 6 8.71% 96.22% 10320 301593 sun.nio.ch.FileChannelImpl.transferTo0 7 0.48% 96.70% 569 301524 org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise 8 0.40% 97.10% 471 301388 sun.nio.ch.EPollArrayWrapper.epollWait 9 0.31% 97.41% 364 301595 sun.nio.ch.FileDispatcherImpl.write0 10 0.29% 97.69% 339 301648 java.io.FileInputStream.readBytes Workload Java samples: Datanode Java samples:
ASIA 2013 (LCA13) www.linaro.org Plain TestDFSIO read tests – More Samples CPU SAMPLES BEGIN (total = 92048) rank self accum count trace method 1 24.81% 24.81% 22833 301222 sun.nio.ch.EPollArrayWrapper.epollWait 2 24.68% 49.48% 22715 301361 sun.nio.ch.EPollArrayWrapper.epollWait 3 24.68% 74.16% 22713 301362 sun.nio.ch.EPollArrayWrapper.epollWait 4 24.67% 98.83% 22708 301360 sun.nio.ch.EPollArrayWrapper.epollWait 5 0.16% 98.99% 147 300075 java.lang.ClassLoader.defineClass1 6 0.05% 99.04% 47 300084 java.util.zip.ZipFile.getEntry 7 0.03% 99.07% 32 300078 java.util.zip.Inflater.inflateBytes 8 0.03% 99.11% 30 300123 java.util.zip.ZipFile.read 9 0.02% 99.13% 21 301170 sun.nio.ch.FileDispatcherImpl.force0 10 0.02% 99.15% 16 301165 org.apache.hadoop.hdfs.server.namenode.EditLogFileOutputStream.<clinit> + 36.37% java libhadoop.so.1.0.0 [.] crc32c_sb8 + 9.40% java [kernel.kallsyms] [k] __copy_to_user_std + 6.02% java perf-14362.map [.] 0xb40d5040 + 5.52% java perf-15039.map [.] 0xb440aba0 + 4.92% java libc-2.15.so [.] memcpy + 2.09% java [kernel.kallsyms] [k] kmap_high + 1.51% java [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore + 1.10% swapper [kernel.kallsyms] [k] default_idle + 0.70% kswapd0 [kernel.kallsyms] [k] __remove_mapping + 0.68% java [kernel.kallsyms] [k] get_page_from_freelist Namenode Java samples: Linux perf results:
ASIA 2013 (LCA13) www.linaro.org New TestDFSIO write tests – Some Samples CPU SAMPLES BEGIN (total = 52774) rank self accum count trace method 1 38.67% 38.67% 20407 300839 sun.nio.ch.EPollArrayWrapper.epollWait 2 32.56% 71.23% 17184 301074 sun.nio.ch.EPollArrayWrapper.epollWait 3 11.51% 82.74% 6076 301070 org.apache.hadoop.util.HyperCrc32C.nativeUpdate 4 4.14% 86.88% 2184 301071 sun.misc.Unsafe.copyMemory 5 2.69% 89.57% 1419 301076 sun.nio.ch.FileDispatcherImpl.write0 6 0.65% 90.22% 341 301102 sun.nio.ch.FileDispatcherImpl.read0 7 0.45% 90.67% 240 301143 java.lang.System.arraycopy 8 0.44% 91.11% 231 300804 sun.nio.ch.EPollArrayWrapper.epollWait 9 0.38% 91.49% 203 301154 sun.nio.ch.EPollArrayWrapper.epollCtl 10 0.38% 91.88% 202 301080 org.apache.hadoop.hdfs.DFSOutputStream.writeChunk CPU SAMPLES BEGIN (total = 234447) rank self accum count trace method 1 17.80% 17.80% 41734 301113 sun.nio.ch.EPollArrayWrapper.epollWait 2 17.77% 35.58% 41672 301307 sun.nio.ch.EPollArrayWrapper.epollWait 3 17.76% 53.33% 41629 301365 sun.nio.ch.EPollArrayWrapper.epollWait 4 17.76% 71.09% 41628 301368 sun.nio.ch.EPollArrayWrapper.epollWait 5 17.75% 88.84% 41621 301364 sun.nio.ch.ServerSocketChannelImpl.accept0 6 3.33% 92.17% 7796 301634 sun.nio.ch.FileDispatcherImpl.write0 7 2.73% 94.89% 6390 301657 java.io.FileOutputStream.writeBytes 8 1.33% 96.22% 3113 301640 sun.nio.ch.FileDispatcherImpl.read0 9 1.30% 97.52% 3038 301631 org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums 10 0.33% 97.84% 766 301641 sun.nio.ch.FileDispatcherImpl.write0 Workload Java samples: Datanode Java samples:
ASIA 2013 (LCA13) www.linaro.org New TestDFSIO write tests – More Samples CPU SAMPLES BEGIN (total = 180508) rank self accum count trace method 1 24.87% 24.87% 44890 301198 sun.nio.ch.EPollArrayWrapper.epollWait 2 24.82% 49.69% 44808 301337 sun.nio.ch.EPollArrayWrapper.epollWait 3 24.82% 74.51% 44806 301338 sun.nio.ch.EPollArrayWrapper.epollWait 4 24.81% 99.33% 44792 301334 sun.nio.ch.EPollArrayWrapper.epollWait 5 0.07% 99.40% 135 300075 java.lang.ClassLoader.defineClass1 6 0.03% 99.43% 46 300080 java.util.zip.ZipFile.getEntry 7 0.02% 99.45% 41 300079 java.util.zip.Inflater.inflateBytes 8 0.01% 99.47% 27 300548 java.io.FileInputStream.readBytes 9 0.01% 99.48% 24 301230 java.lang.UNIXProcess.waitForProcessExit 10 0.01% 99.49% 22 301145 sun.nio.ch.FileDispatcherImpl.force0 + 18.04% java libhadoop.so.1.0.0 [.] fold + 14.38% java perf-16729.map [.] 0xb42c0840 + 6.45% java libc-2.15.so [.] memcpy + 5.77% java [kernel.kallsyms] [k] __copy_from_user + 3.89% java [kernel.kallsyms] [k] __copy_to_user_std + 2.73% java perf-16647.map [.] 0xb4142560 + 2.42% java [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore + 1.91% java libhadoop.so.1.0.0 [.] crc32c_sb8 + 1.28% swapper [kernel.kallsyms] [k] finish_task_switch + 0.96% swapper [kernel.kallsyms] [k] tick_nohz_idle_enter + 0.88% java libhadoop.so.1.0.0 [.] crc32c_neon Namenode Java samples: Linux perf results:
ASIA 2013 (LCA13) www.linaro.org New TestDFSIO read tests – Some Samples CPU SAMPLES BEGIN (total = 13997) rank self accum count trace method 1 41.44% 41.44% 5800 301067 sun.nio.ch.EPollArrayWrapper.epollWait 2 25.40% 66.84% 3555 301075 org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums 3 13.15% 79.98% 1840 301074 sun.nio.ch.FileDispatcherImpl.read0 4 4.63% 84.61% 648 301076 sun.misc.Unsafe.copyMemory 5 2.22% 86.83% 311 301085 sun.nio.ch.SocketChannelImpl.isConnected 6 1.86% 88.70% 261 301077 sun.nio.ch.FileDispatcherImpl.read0 7 1.56% 90.26% 219 300827 sun.nio.ch.EPollArrayWrapper.epollWait 8 1.15% 91.41% 161 300790 sun.nio.ch.EPollArrayWrapper.epollWait 9 0.64% 92.05% 89 300072 java.lang.ClassLoader.defineClass1 10 0.61% 92.66% 86 301099 sun.nio.ch.SocketChannelImpl.isConnected CPU SAMPLES BEGIN (total = 123308) rank self accum count trace method 1 17.64% 17.64% 21751 301117 sun.nio.ch.EPollArrayWrapper.epollWait 2 17.59% 35.23% 21686 301307 sun.nio.ch.EPollArrayWrapper.epollWait 3 17.55% 52.78% 21643 301370 sun.nio.ch.EPollArrayWrapper.epollWait 4 17.55% 70.33% 21643 301371 sun.nio.ch.EPollArrayWrapper.epollWait 5 17.55% 87.88% 21640 301369 sun.nio.ch.ServerSocketChannelImpl.accept0 6 8.05% 95.93% 9930 301604 sun.nio.ch.FileChannelImpl.transferTo0 7 0.78% 96.71% 963 301567 java.io.FileInputStream.readBytes 8 0.49% 97.20% 604 301580 org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise 9 0.26% 97.46% 320 301607 sun.nio.ch.FileDispatcherImpl.write0 10 0.22% 97.68% 268 301586 sun.nio.ch.EPollArrayWrapper.epollWait Workload Java samples: Datanode Java samples:
ASIA 2013 (LCA13) www.linaro.org New TestDFSIO read tests – More Samples CPU SAMPLES BEGIN (total = 97523) rank self accum count trace method 1 24.83% 24.83% 24218 301190 sun.nio.ch.EPollArrayWrapper.epollWait 2 24.72% 49.55% 24103 301336 sun.nio.ch.EPollArrayWrapper.epollWait 3 24.71% 74.26% 24101 301337 sun.nio.ch.EPollArrayWrapper.epollWait 4 24.71% 98.97% 24096 301335 sun.nio.ch.EPollArrayWrapper.epollWait 5 0.14% 99.11% 133 300073 java.lang.ClassLoader.defineClass1 6 0.05% 99.16% 50 300085 java.util.zip.ZipFile.getEntry 7 0.05% 99.20% 45 300078 java.util.zip.Inflater.inflateBytes 8 0.02% 99.22% 21 300081 java.util.zip.ZipFile.read 9 0.02% 99.25% 21 301135 sun.nio.ch.FileDispatcherImpl.force0 10 0.01% 99.26% 13 301128 org.apache.hadoop.hdfs.server.namenode.EditLogFileOutputStream.<clinit> + 20.63% java libhadoop.so.1.0.0 [.] fold + 10.89% java [kernel.kallsyms] [k] __copy_to_user_std + 6.58% java perf-15890.map [.] 0xb44417f8 + 5.74% java perf-15818.map [.] 0xb41a8de0 + 5.37% java libc-2.15.so [.] memcpy + 3.15% java libhadoop.so.1.0.0 [.] crc32c_sb8 + 2.64% java [kernel.kallsyms] [k] kmap_high + 2.01% java [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore + 1.31% swapper [kernel.kallsyms] [k] default_idle + 1.30% java libhadoop.so.1.0.0 [.] crc32c_neon + 0.88% java [kernel.kallsyms] [k] get_page_from_freelist Namenode Java samples: Linux perf results:
More about Linaro Connect: www.linaro.org/connect/ More about Linaro: www.linaro.org/about/ More about Linaro engineering: www.linaro.org/engineering/ ASIA 2013 (LCA13)

More Related Content

PDF
BKK16-408B Data Analytics and Machine Learning From Node to Cluster
Linaro
 
PDF
BKK16-404B Data Analytics and Machine Learning- from Node to Cluster
Linaro
 
PDF
Matrix Factorizations at Scale: a Comparison of Scientific Data Analytics on ...
Databricks
 
PDF
Flare: Scale Up Spark SQL with Native Compilation and Set Your Data on Fire! ...
Databricks
 
PDF
Apache Spark: The Next Gen toolset for Big Data Processing
prajods
 
PPT
Hadoop for Scientific Workloads__HadoopSummit2010
Yahoo Developer Network
 
PDF
Spark on yarn
datamantra
 
PDF
Leveraging GPU-Accelerated Analytics on top of Apache Spark with Todd Mostak
Databricks
 
BKK16-408B Data Analytics and Machine Learning From Node to Cluster
Linaro
 
BKK16-404B Data Analytics and Machine Learning- from Node to Cluster
Linaro
 
Matrix Factorizations at Scale: a Comparison of Scientific Data Analytics on ...
Databricks
 
Flare: Scale Up Spark SQL with Native Compilation and Set Your Data on Fire! ...
Databricks
 
Apache Spark: The Next Gen toolset for Big Data Processing
prajods
 
Hadoop for Scientific Workloads__HadoopSummit2010
Yahoo Developer Network
 
Spark on yarn
datamantra
 
Leveraging GPU-Accelerated Analytics on top of Apache Spark with Todd Mostak
Databricks
 

What's hot (20)

PDF
A Spark Framework For &lt; $100, &lt; 1 Hour, Accurate Personalized DNA Analy...
Spark Summit
 
PDF
myHadoop 0.30
Glenn K. Lockwood
 
PDF
Spark Based Distributed Deep Learning Framework For Big Data Applications
Humoyun Ahmedov
 
PDF
Distributed Deep Learning with Apache Spark and TensorFlow with Jim Dowling
Databricks
 
PDF
Deep Learning with DL4J on Apache Spark: Yeah it’s Cool, but are You Doing it...
Databricks
 
PPTX
Introduction to Apache Spark
Hubert Fan Chiang
 
PPT
Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive
Sachin Aggarwal
 
PPTX
Parallel Linear Regression in Interative Reduce and YARN
DataWorks Summit
 
PDF
Re-Architecting Spark For Performance Understandability
Jen Aman
 
PDF
Hadoop Spark Introduction-20150130
Xuan-Chao Huang
 
PPTX
Deep Learning with Spark and GPUs
DataWorks Summit
 
PPTX
Introduction to Apache Spark Developer Training
Cloudera, Inc.
 
PDF
Advanced Data Science on Spark-(Reza Zadeh, Stanford)
Spark Summit
 
PPTX
Let Spark Fly: Advantages and Use Cases for Spark on Hadoop
MapR Technologies
 
PDF
Hadoop and Spark
Shravan (Sean) Pabba
 
PDF
Marcel Kornacker: Impala tech talk Tue Feb 26th 2013
Modern Data Stack France
 
PDF
Lambda Architecture using Google Cloud plus Apps
Simon Su
 
PPTX
Advanced spark deep learning
Adam Gibson
 
PDF
Introduction to Apache Spark Ecosystem
Bojan Babic
 
PPTX
Apache Spark Architecture
Alexey Grishchenko
 
A Spark Framework For &lt; $100, &lt; 1 Hour, Accurate Personalized DNA Analy...
Spark Summit
 
myHadoop 0.30
Glenn K. Lockwood
 
Spark Based Distributed Deep Learning Framework For Big Data Applications
Humoyun Ahmedov
 
Distributed Deep Learning with Apache Spark and TensorFlow with Jim Dowling
Databricks
 
Deep Learning with DL4J on Apache Spark: Yeah it’s Cool, but are You Doing it...
Databricks
 
Introduction to Apache Spark
Hubert Fan Chiang
 
Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive
Sachin Aggarwal
 
Parallel Linear Regression in Interative Reduce and YARN
DataWorks Summit
 
Re-Architecting Spark For Performance Understandability
Jen Aman
 
Hadoop Spark Introduction-20150130
Xuan-Chao Huang
 
Deep Learning with Spark and GPUs
DataWorks Summit
 
Introduction to Apache Spark Developer Training
Cloudera, Inc.
 
Advanced Data Science on Spark-(Reza Zadeh, Stanford)
Spark Summit
 
Let Spark Fly: Advantages and Use Cases for Spark on Hadoop
MapR Technologies
 
Hadoop and Spark
Shravan (Sean) Pabba
 
Marcel Kornacker: Impala tech talk Tue Feb 26th 2013
Modern Data Stack France
 
Lambda Architecture using Google Cloud plus Apps
Simon Su
 
Advanced spark deep learning
Adam Gibson
 
Introduction to Apache Spark Ecosystem
Bojan Babic
 
Apache Spark Architecture
Alexey Grishchenko
 
Ad

Similar to LCA13: Hadoop DFS Performance (20)

PPTX
Hadoop Vectored IO
Steve Loughran
 
PDF
Resource planning on the (Amazon) cloud
Enis Afgan
 
PDF
Testing Delphix: easy data virtualization
Franck Pachot
 
PDF
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Odinot Stanislas
 
PDF
Ceph
Hien Nguyen Van
 
PPTX
Introduction to Hadoop part 2
Giovanna Roda
 
PDF
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...
Flink Forward
 
ODP
Experience In Building Scalable Web Sites Through Infrastructure's View
Phuwadon D
 
PDF
Demo 0.9.4
eTimeline, LLC
 
PDF
Spark Summit EU 2015: Lessons from 300+ production users
Databricks
 
PDF
Direct Code Execution - LinuxCon Japan 2014
Hajime Tazaki
 
PDF
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
InfluxData
 
PPTX
How swift is your Swift - SD.pptx
OpenStack Foundation
 
PDF
Java File I/O Performance Analysis - Part I - JCConf 2018
Michael Fong
 
PDF
DrupalCampLA 2011: Drupal backend-performance
Ashok Modi
 
ODP
Nagios Conference 2012 - Dan Wittenberg - Case Study: Scaling Nagios Core at ...
Nagios
 
PDF
Node Interactive Debugging Node.js In Production
Yunong Xiao
 
PPTX
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...
Виталий Стародубцев
 
PDF
Resource-Efficient Deep Learning Model Selection on Apache Spark
Databricks
 
PDF
OpenNebulaConf 2016 - Measuring and tuning VM performance by Boyan Krosnov, S...
OpenNebula Project
 
Hadoop Vectored IO
Steve Loughran
 
Resource planning on the (Amazon) cloud
Enis Afgan
 
Testing Delphix: easy data virtualization
Franck Pachot
 
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Odinot Stanislas
 
Ceph
Hien Nguyen Van
 
Introduction to Hadoop part 2
Giovanna Roda
 
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...
Flink Forward
 
Experience In Building Scalable Web Sites Through Infrastructure's View
Phuwadon D
 
Demo 0.9.4
eTimeline, LLC
 
Spark Summit EU 2015: Lessons from 300+ production users
Databricks
 
Direct Code Execution - LinuxCon Japan 2014
Hajime Tazaki
 
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
InfluxData
 
How swift is your Swift - SD.pptx
OpenStack Foundation
 
Java File I/O Performance Analysis - Part I - JCConf 2018
Michael Fong
 
DrupalCampLA 2011: Drupal backend-performance
Ashok Modi
 
Nagios Conference 2012 - Dan Wittenberg - Case Study: Scaling Nagios Core at ...
Nagios
 
Node Interactive Debugging Node.js In Production
Yunong Xiao
 
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...
Виталий Стародубцев
 
Resource-Efficient Deep Learning Model Selection on Apache Spark
Databricks
 
OpenNebulaConf 2016 - Measuring and tuning VM performance by Boyan Krosnov, S...
OpenNebula Project
 
Ad

More from Linaro (20)

PDF
Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Linaro
 
PDF
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Linaro
 
PDF
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Linaro
 
PDF
Bud17 113: distribution ci using qemu and open qa
Linaro
 
PDF
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
Linaro
 
PDF
HPC network stack on ARM - Linaro HPC Workshop 2018
Linaro
 
PDF
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
Linaro
 
PDF
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Linaro
 
PDF
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Linaro
 
PDF
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Linaro
 
PDF
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
Linaro
 
PDF
HKG18-100K1 - George Grey: Opening Keynote
Linaro
 
PDF
HKG18-318 - OpenAMP Workshop
Linaro
 
PDF
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
Linaro
 
PDF
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
Linaro
 
PDF
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
Linaro
 
PDF
HKG18-TR08 - Upstreaming SVE in QEMU
Linaro
 
PDF
HKG18-113- Secure Data Path work with i.MX8M
Linaro
 
PPTX
HKG18-120 - Devicetree Schema Documentation and Validation
Linaro
 
PPTX
HKG18-223 - Trusted FirmwareM: Trusted boot
Linaro
 
Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Linaro
 
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Linaro
 
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Linaro
 
Bud17 113: distribution ci using qemu and open qa
Linaro
 
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
Linaro
 
HPC network stack on ARM - Linaro HPC Workshop 2018
Linaro
 
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
Linaro
 
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Linaro
 
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Linaro
 
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Linaro
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
Linaro
 
HKG18-100K1 - George Grey: Opening Keynote
Linaro
 
HKG18-318 - OpenAMP Workshop
Linaro
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
Linaro
 
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
Linaro
 
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
Linaro
 
HKG18-TR08 - Upstreaming SVE in QEMU
Linaro
 
HKG18-113- Secure Data Path work with i.MX8M
Linaro
 
HKG18-120 - Devicetree Schema Documentation and Validation
Linaro
 
HKG18-223 - Trusted FirmwareM: Trusted boot
Linaro
 

Recently uploaded (20)

PDF
Advanced IT Governance
University of Hertfordshire
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
ssuserf8b8bd1
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
AbdullahSani29
 
Advanced IT Governance
University of Hertfordshire
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
ssuserf8b8bd1
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
AbdullahSani29
 

LCA13: Hadoop DFS Performance

  • 1. ASIA 2013 (LCA13) LEG – Hadoop DFS Performance Steve Capper <steve.capper@linaro.org>
  • 2. ASIA 2013 (LCA13) www.linaro.org Hadoop Performance on ARM I concentrated on reducing the time taken for CPU bound tasks (the latency). My work, so far, has focussed on the underlying cluster filesystem, HDFS, as this underpins a lot of Hadoop workloads. The latest release-tagged Hadoop at the time of experimentation was 2.0.2-alpha. So this was the version used. (2.0.3-alpha has just come out). Hadoop installations consist of rather a lot of moving parts. So please let me know if I should be concentrating on something else! :-)
  • 3. ASIA 2013 (LCA13) www.linaro.org Hadoop Distributed Filesystem – Overview HDFS is the default Hadoop distributed filesystem. Consists of multiple “nodes”: Namenode – holds the metadata for the filesystem and keeps track of the datanodes. Only one namenode is active at a time per namespace. Thus there is a high memory requirement for a namenode. Datanodes – store the file's blocks. The default block size is 64MB. (Optional) Passive namenodes – To maintain an High Availability configuration, one can set up shared storage (via NFS or by journalnodes) and have namenodes on standby to failover. Filesystem blocks are replicated between datanodes. Datanodes can be “rack aware”; data will be distributed between racks. Nodes can run on the same machine or on different machines.
  • 4. ASIA 2013 (LCA13) www.linaro.org Data Integrity in HDFS Hadoop mitigates against hardware failure in the following ways: Metadata can be saved to multiple filesystems and regular snapshots are usually taken to allow for disaster recovery. On some HA configurations, multiple journalnodes maintain a quorum of the metadata. Data blocks are replicated across multiple datanodes (preferably to different racks). Data blocks are regularly transmitted: All data streams are checksummed. The default checksum algorithm is CRC32c. The default number of bytes per checksum is 512 bytes.
  • 5. ASIA 2013 (LCA13) www.linaro.org Test Configuration So far I have been micro-benchmarking single machine workloads (1 name node, 1 data node, 1 workload). I have been working on an A9 platform. TestDFSIO -read and -write have been tested with a 10GB filesize. Soft-float Oracle JDK 1.7.0_10 was used. Performance was measured using Linux perf -a, Java samples were taken with the inbuilt profiler: -Xrunhprof:cpu=samples,depth=10
  • 6. ASIA 2013 (LCA13) www.linaro.org Plain TestDFSIO write tests – Samples CPU SAMPLES BEGIN (total = 164879) rank self accum count trace method 1 24.85% 24.85% 40970 301219 sun.nio.ch.EPollArrayWrapper.epollWait 2 24.79% 49.64% 40870 301362 sun.nio.ch.EPollArrayWrapper.epollWait 3 24.79% 74.42% 40866 301363 sun.nio.ch.EPollArrayWrapper.epollWait 4 24.78% 99.20% 40859 301358 sun.nio.ch.EPollArrayWrapper.epollWait 5 0.07% 99.27% 118 300073 java.lang.ClassLoader.defineClass1 6 0.03% 99.31% 57 300020 java.util.zip.ZipFile.open 7 0.03% 99.34% 55 300179 java.util.zip.ZipFile.getEntry 8 0.03% 99.38% 54 300062 java.util.zip.ZipFile.read 9 0.02% 99.39% 31 300076 java.util.zip.Inflater.inflateBytes 10 0.01% 99.41% 23 301165 sun.nio.ch.FileDispatcherImpl.force0 + 33.80% java perf-13641.map [.] 0xb4258538 + 19.98% java perf-13534.map [.] 0xb403e980 + 5.37% java libc-2.15.so [.] memcpy + 3.02% java [kernel.kallsyms] [k] __copy_from_user + 2.60% java [kernel.kallsyms] [k] __copy_to_user_std + 1.86% java [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore + 1.01% swapper [kernel.kallsyms] [k] finish_task_switch + 0.66% swapper [kernel.kallsyms] [k] tick_nohz_idle_enter + 0.55% java [kernel.kallsyms] [k] get_page_from_freelist + 0.44% java [kernel.kallsyms] [k] free_hot_cold_page + 0.42% swapper [kernel.kallsyms] [k] default_idle Namenode Java samples: Linux perf results:
  • 7. ASIA 2013 (LCA13) www.linaro.org Plain TestDFSIO read & write CPU usage Plain TestDFSIO Write Plain TestDFSIO Read Other Java memcpy crc32c_sb8
  • 8. ASIA 2013 (LCA13) www.linaro.org Some analysis of plain TestDFSIO The namenode doesn't do much when it's in charge of 1 datanode. (this will change when things scale up). When writing data: Most of the Java time is spent in PureJavaCrc32C.update Most of the CPU time is spent in Java land. When reading data: Most of the Java time is spent in NativeCrc32.nativeVerifyChunkedSums. Most of the CPU time is in the native function: crc32c_sb8 For both reading and writing data: ~10% of the CPU time is spent copying memory.
  • 9. ASIA 2013 (LCA13) www.linaro.org Optimising the read and write paths Trevor Robinson has proposed a patch to improve the write path: HDFS-3529 “Use direct buffers for data in write path” I have been working on speeding up the computation of CRC32c checksums, by using NEON. Also, I've performed some very preliminary experiments on replacing PureJavaCrc32C with a JNI class that references the NEON CRC code.
  • 10. ASIA 2013 (LCA13) www.linaro.org NEON optimisation of the CRC32c algorithm I worked on an algorithm Q4 last year: https://wiki.linaro.org/LEG/Engineering/CRC Given an input buffer, we perform polynomial multiplication and addition to give a slighter smaller buffer with the same CRC. This is referred to as “folding”. The algorithm reduced the buffer by 32 bytes at a time. The final buffer of 32 bytes was then processed by slice-by-8 as normal. I found that the vast majority of CRCs in Hadoop (in both NativeCrc32.nativeVerifyChunkedSums and PureJavaCrc32C.update) were computed for 512 byte buffers.
  • 11. ASIA 2013 (LCA13) www.linaro.org A single 64 bit fold 32 64 64 M(x)32 A(x)B(x) 64 =A(x) * (x^64 mod P(x)) =B(x) * (x^96 mod P(x)) M'(x) + + 64 CRC(M'(x)) = CRC(M(x)) =
  • 12. ASIA 2013 (LCA13) www.linaro.org NEON implementations With ARMv7 NEON we can only perform polynomial multiplication on 8 bit lanes: We need to be able to multiply at least 32 bits, thus multiple vmull.p8's were chained together to achieve this). There are 16 vmull.p8's per fold, so a little register pressure! A gcc intrinsic version has been coded up: This is considerably simpler to look at. Unfortunately, it runs a little slower than hand-optimised assembler. A test case has been sent to the Linaro Toolchain Working group (a couple of weeks ago) and this is being analysed by them.
  • 13. ASIA 2013 (LCA13) www.linaro.org Replacing PureJavaCrc32C PureJavaCrc32C has two noteworthy methods: public void update(byte[] b, int off, int len) – called mostly for lengths of 512. public void update(int b) – seldom called. I created a new class implementing the Checksum interface that: Used the same implementation of: update(int b) as PureJavaCrc32C. Called straight into JNI for update(byte[] b, int off, int len). The class name NativeCrc32 was already taken by something else, so I chose a rather silly temporary name: HyperCrc32C
  • 14. ASIA 2013 (LCA13) www.linaro.org Dealing with byte [] in JNI We are only reading from the byte[] array, and only for a very short time. Thus I pinned the buffer in memory with: GetPrimitiveArrayCritical Then subsequently released the buffer with ReleasePrimitiveArrayCritical(..., JNI_ABORT) This worked for me, but perhaps a better long term solution would be to change the backing data type to a ByteBuffer? We could also be clever and change the alignment of these? Rather than test every optimisation individually in this talk; I am going to put them all together.
  • 15. ASIA 2013 (LCA13) www.linaro.org TestDFSIO read & write CPU usage Plain TestDFSIO Write New TestDFSIO Write Other Java memcpy crc Plain TestDFSIO Read New TestDFSIO Read
  • 16. ASIA 2013 (LCA13) www.linaro.org Some Analysis of the New TestDFSIO runs The name node samples are unchanged. For the write path: Most of the time is now spent running native code rather than Java code. There is a noticeable reduction in Hadoop user CPU usage. For the read path: Most of the time is still spent running native code. There is again a reduction in Hadoop user CPU usage. For both the read and write paths there is a significant amount of CPU time spent copying memory around.
  • 17. ASIA 2013 (LCA13) www.linaro.org Conclusions and Further Work The CPU usage required for TestDFSIO runs has been reduced for both read/write paths. This gives us more CPU cycles to run Hadoop jobs with! Hadoop is known to be very sensitive to underlying disk IO. To optimise HDFS IO, it would make sense to optimise disk/filesystem IO as much as possible and re-run these benchmarks. As Hadoop runs under Java: It makes sense to keep track of JVMs. A beta hard-float JVM has been released. The CPU usage for memcpy is making me uneasy!
  • 18. ASIA 2013 (LCA13) www.linaro.org We should also consider our hardware... HDD SSD 0 0.5 1 1.5 2 2.5 TestDFSIO Write Relative performance MB/s HDD SSD 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 TestDFSIO Read Relative performance MB/s
  • 19. ASIA 2013 (LCA13) www.linaro.org Plain TestDFSIO write tests – Some Samples CPU SAMPLES BEGIN (total = 81097) rank self accum count trace method 1 36.88% 36.88% 29906 300854 sun.nio.ch.EPollArrayWrapper.epollWait 2 32.95% 69.83% 26723 301107 sun.nio.ch.EPollArrayWrapper.epollWait 3 16.48% 86.31% 13363 301109 org.apache.hadoop.util.PureJavaCrc32C.update 4 2.74% 89.05% 2225 301113 sun.misc.Unsafe.copyMemory 5 1.67% 90.72% 1357 301115 sun.nio.ch.FileDispatcherImpl.write0 6 1.09% 91.81% 884 301128 org.apache.hadoop.hdfs.DFSOutputStream.writeChunk 7 0.70% 92.51% 564 301110 sun.nio.ch.FileDispatcherImpl.read0 8 0.42% 92.93% 342 301123 org.apache.hadoop.hdfs.DFSOutputStream.writeChunk 9 0.40% 93.33% 326 301202 org.apache.hadoop.hdfs.DFSOutputStream.waitAndQueueCurrentPacket 10 0.33% 93.66% 266 301124 java.lang.System.arraycopy CPU SAMPLES BEGIN (total = 227245) rank self accum count trace method 1 16.85% 16.85% 38293 301110 sun.nio.ch.EPollArrayWrapper.epollWait 2 16.82% 33.67% 38228 301313 sun.nio.ch.EPollArrayWrapper.epollWait 3 16.80% 50.48% 38184 301374 sun.nio.ch.EPollArrayWrapper.epollWait 4 16.80% 67.28% 38184 301375 sun.nio.ch.EPollArrayWrapper.epollWait 5 16.80% 84.08% 38169 301373 sun.nio.ch.ServerSocketChannelImpl.accept0 6 6.75% 90.83% 15342 301646 java.io.FileOutputStream.writeBytes 7 2.77% 93.60% 6294 301642 org.apache.hadoop.util.DataChecksum.verifyChunkedSums 8 2.49% 96.08% 5653 301664 java.io.FileOutputStream.writeBytes 9 1.14% 97.22% 2587 301640 sun.nio.ch.FileDispatcherImpl.read0 10 0.35% 97.57% 798 301691 sun.nio.ch.FileDispatcherImpl.write0 Workload Java samples: Datanode Java samples:
  • 20. ASIA 2013 (LCA13) www.linaro.org Plain TestDFSIO read tests – Some Samples CPU SAMPLES BEGIN (total = 13668) rank self accum count trace method 1 36.82% 36.82% 5033 301083 org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums 2 31.37% 68.19% 4287 301080 sun.nio.ch.EPollArrayWrapper.epollWait 3 14.00% 82.18% 1913 301086 sun.nio.ch.FileDispatcherImpl.read0 4 4.82% 87.01% 659 301082 sun.misc.Unsafe.copyMemory 5 2.00% 89.00% 273 301118 sun.nio.ch.SocketChannelImpl.isConnected 6 1.87% 90.88% 256 301089 sun.nio.ch.FileDispatcherImpl.read0 7 0.76% 91.64% 104 300072 java.lang.ClassLoader.defineClass1 8 0.61% 92.25% 84 301105 sun.nio.ch.SocketChannelImpl.isConnected 9 0.51% 92.76% 70 300841 sun.nio.ch.EPollArrayWrapper.epollWait 10 0.49% 93.25% 67 301127 sun.misc.Unsafe.copyMemory CPU SAMPLES BEGIN (total = 118437) rank self accum count trace method 1 17.57% 17.57% 20810 301107 sun.nio.ch.EPollArrayWrapper.epollWait 2 17.51% 35.08% 20741 301304 sun.nio.ch.EPollArrayWrapper.epollWait 3 17.48% 52.56% 20697 301368 sun.nio.ch.EPollArrayWrapper.epollWait 4 17.48% 70.03% 20697 301369 sun.nio.ch.EPollArrayWrapper.epollWait 5 17.47% 87.51% 20695 301367 sun.nio.ch.ServerSocketChannelImpl.accept0 6 8.71% 96.22% 10320 301593 sun.nio.ch.FileChannelImpl.transferTo0 7 0.48% 96.70% 569 301524 org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise 8 0.40% 97.10% 471 301388 sun.nio.ch.EPollArrayWrapper.epollWait 9 0.31% 97.41% 364 301595 sun.nio.ch.FileDispatcherImpl.write0 10 0.29% 97.69% 339 301648 java.io.FileInputStream.readBytes Workload Java samples: Datanode Java samples:
  • 21. ASIA 2013 (LCA13) www.linaro.org Plain TestDFSIO read tests – More Samples CPU SAMPLES BEGIN (total = 92048) rank self accum count trace method 1 24.81% 24.81% 22833 301222 sun.nio.ch.EPollArrayWrapper.epollWait 2 24.68% 49.48% 22715 301361 sun.nio.ch.EPollArrayWrapper.epollWait 3 24.68% 74.16% 22713 301362 sun.nio.ch.EPollArrayWrapper.epollWait 4 24.67% 98.83% 22708 301360 sun.nio.ch.EPollArrayWrapper.epollWait 5 0.16% 98.99% 147 300075 java.lang.ClassLoader.defineClass1 6 0.05% 99.04% 47 300084 java.util.zip.ZipFile.getEntry 7 0.03% 99.07% 32 300078 java.util.zip.Inflater.inflateBytes 8 0.03% 99.11% 30 300123 java.util.zip.ZipFile.read 9 0.02% 99.13% 21 301170 sun.nio.ch.FileDispatcherImpl.force0 10 0.02% 99.15% 16 301165 org.apache.hadoop.hdfs.server.namenode.EditLogFileOutputStream.<clinit> + 36.37% java libhadoop.so.1.0.0 [.] crc32c_sb8 + 9.40% java [kernel.kallsyms] [k] __copy_to_user_std + 6.02% java perf-14362.map [.] 0xb40d5040 + 5.52% java perf-15039.map [.] 0xb440aba0 + 4.92% java libc-2.15.so [.] memcpy + 2.09% java [kernel.kallsyms] [k] kmap_high + 1.51% java [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore + 1.10% swapper [kernel.kallsyms] [k] default_idle + 0.70% kswapd0 [kernel.kallsyms] [k] __remove_mapping + 0.68% java [kernel.kallsyms] [k] get_page_from_freelist Namenode Java samples: Linux perf results:
  • 22. ASIA 2013 (LCA13) www.linaro.org New TestDFSIO write tests – Some Samples CPU SAMPLES BEGIN (total = 52774) rank self accum count trace method 1 38.67% 38.67% 20407 300839 sun.nio.ch.EPollArrayWrapper.epollWait 2 32.56% 71.23% 17184 301074 sun.nio.ch.EPollArrayWrapper.epollWait 3 11.51% 82.74% 6076 301070 org.apache.hadoop.util.HyperCrc32C.nativeUpdate 4 4.14% 86.88% 2184 301071 sun.misc.Unsafe.copyMemory 5 2.69% 89.57% 1419 301076 sun.nio.ch.FileDispatcherImpl.write0 6 0.65% 90.22% 341 301102 sun.nio.ch.FileDispatcherImpl.read0 7 0.45% 90.67% 240 301143 java.lang.System.arraycopy 8 0.44% 91.11% 231 300804 sun.nio.ch.EPollArrayWrapper.epollWait 9 0.38% 91.49% 203 301154 sun.nio.ch.EPollArrayWrapper.epollCtl 10 0.38% 91.88% 202 301080 org.apache.hadoop.hdfs.DFSOutputStream.writeChunk CPU SAMPLES BEGIN (total = 234447) rank self accum count trace method 1 17.80% 17.80% 41734 301113 sun.nio.ch.EPollArrayWrapper.epollWait 2 17.77% 35.58% 41672 301307 sun.nio.ch.EPollArrayWrapper.epollWait 3 17.76% 53.33% 41629 301365 sun.nio.ch.EPollArrayWrapper.epollWait 4 17.76% 71.09% 41628 301368 sun.nio.ch.EPollArrayWrapper.epollWait 5 17.75% 88.84% 41621 301364 sun.nio.ch.ServerSocketChannelImpl.accept0 6 3.33% 92.17% 7796 301634 sun.nio.ch.FileDispatcherImpl.write0 7 2.73% 94.89% 6390 301657 java.io.FileOutputStream.writeBytes 8 1.33% 96.22% 3113 301640 sun.nio.ch.FileDispatcherImpl.read0 9 1.30% 97.52% 3038 301631 org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums 10 0.33% 97.84% 766 301641 sun.nio.ch.FileDispatcherImpl.write0 Workload Java samples: Datanode Java samples:
  • 23. ASIA 2013 (LCA13) www.linaro.org New TestDFSIO write tests – More Samples CPU SAMPLES BEGIN (total = 180508) rank self accum count trace method 1 24.87% 24.87% 44890 301198 sun.nio.ch.EPollArrayWrapper.epollWait 2 24.82% 49.69% 44808 301337 sun.nio.ch.EPollArrayWrapper.epollWait 3 24.82% 74.51% 44806 301338 sun.nio.ch.EPollArrayWrapper.epollWait 4 24.81% 99.33% 44792 301334 sun.nio.ch.EPollArrayWrapper.epollWait 5 0.07% 99.40% 135 300075 java.lang.ClassLoader.defineClass1 6 0.03% 99.43% 46 300080 java.util.zip.ZipFile.getEntry 7 0.02% 99.45% 41 300079 java.util.zip.Inflater.inflateBytes 8 0.01% 99.47% 27 300548 java.io.FileInputStream.readBytes 9 0.01% 99.48% 24 301230 java.lang.UNIXProcess.waitForProcessExit 10 0.01% 99.49% 22 301145 sun.nio.ch.FileDispatcherImpl.force0 + 18.04% java libhadoop.so.1.0.0 [.] fold + 14.38% java perf-16729.map [.] 0xb42c0840 + 6.45% java libc-2.15.so [.] memcpy + 5.77% java [kernel.kallsyms] [k] __copy_from_user + 3.89% java [kernel.kallsyms] [k] __copy_to_user_std + 2.73% java perf-16647.map [.] 0xb4142560 + 2.42% java [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore + 1.91% java libhadoop.so.1.0.0 [.] crc32c_sb8 + 1.28% swapper [kernel.kallsyms] [k] finish_task_switch + 0.96% swapper [kernel.kallsyms] [k] tick_nohz_idle_enter + 0.88% java libhadoop.so.1.0.0 [.] crc32c_neon Namenode Java samples: Linux perf results:
  • 24. ASIA 2013 (LCA13) www.linaro.org New TestDFSIO read tests – Some Samples CPU SAMPLES BEGIN (total = 13997) rank self accum count trace method 1 41.44% 41.44% 5800 301067 sun.nio.ch.EPollArrayWrapper.epollWait 2 25.40% 66.84% 3555 301075 org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums 3 13.15% 79.98% 1840 301074 sun.nio.ch.FileDispatcherImpl.read0 4 4.63% 84.61% 648 301076 sun.misc.Unsafe.copyMemory 5 2.22% 86.83% 311 301085 sun.nio.ch.SocketChannelImpl.isConnected 6 1.86% 88.70% 261 301077 sun.nio.ch.FileDispatcherImpl.read0 7 1.56% 90.26% 219 300827 sun.nio.ch.EPollArrayWrapper.epollWait 8 1.15% 91.41% 161 300790 sun.nio.ch.EPollArrayWrapper.epollWait 9 0.64% 92.05% 89 300072 java.lang.ClassLoader.defineClass1 10 0.61% 92.66% 86 301099 sun.nio.ch.SocketChannelImpl.isConnected CPU SAMPLES BEGIN (total = 123308) rank self accum count trace method 1 17.64% 17.64% 21751 301117 sun.nio.ch.EPollArrayWrapper.epollWait 2 17.59% 35.23% 21686 301307 sun.nio.ch.EPollArrayWrapper.epollWait 3 17.55% 52.78% 21643 301370 sun.nio.ch.EPollArrayWrapper.epollWait 4 17.55% 70.33% 21643 301371 sun.nio.ch.EPollArrayWrapper.epollWait 5 17.55% 87.88% 21640 301369 sun.nio.ch.ServerSocketChannelImpl.accept0 6 8.05% 95.93% 9930 301604 sun.nio.ch.FileChannelImpl.transferTo0 7 0.78% 96.71% 963 301567 java.io.FileInputStream.readBytes 8 0.49% 97.20% 604 301580 org.apache.hadoop.io.nativeio.NativeIO.posix_fadvise 9 0.26% 97.46% 320 301607 sun.nio.ch.FileDispatcherImpl.write0 10 0.22% 97.68% 268 301586 sun.nio.ch.EPollArrayWrapper.epollWait Workload Java samples: Datanode Java samples:
  • 25. ASIA 2013 (LCA13) www.linaro.org New TestDFSIO read tests – More Samples CPU SAMPLES BEGIN (total = 97523) rank self accum count trace method 1 24.83% 24.83% 24218 301190 sun.nio.ch.EPollArrayWrapper.epollWait 2 24.72% 49.55% 24103 301336 sun.nio.ch.EPollArrayWrapper.epollWait 3 24.71% 74.26% 24101 301337 sun.nio.ch.EPollArrayWrapper.epollWait 4 24.71% 98.97% 24096 301335 sun.nio.ch.EPollArrayWrapper.epollWait 5 0.14% 99.11% 133 300073 java.lang.ClassLoader.defineClass1 6 0.05% 99.16% 50 300085 java.util.zip.ZipFile.getEntry 7 0.05% 99.20% 45 300078 java.util.zip.Inflater.inflateBytes 8 0.02% 99.22% 21 300081 java.util.zip.ZipFile.read 9 0.02% 99.25% 21 301135 sun.nio.ch.FileDispatcherImpl.force0 10 0.01% 99.26% 13 301128 org.apache.hadoop.hdfs.server.namenode.EditLogFileOutputStream.<clinit> + 20.63% java libhadoop.so.1.0.0 [.] fold + 10.89% java [kernel.kallsyms] [k] __copy_to_user_std + 6.58% java perf-15890.map [.] 0xb44417f8 + 5.74% java perf-15818.map [.] 0xb41a8de0 + 5.37% java libc-2.15.so [.] memcpy + 3.15% java libhadoop.so.1.0.0 [.] crc32c_sb8 + 2.64% java [kernel.kallsyms] [k] kmap_high + 2.01% java [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore + 1.31% swapper [kernel.kallsyms] [k] default_idle + 1.30% java libhadoop.so.1.0.0 [.] crc32c_neon + 0.88% java [kernel.kallsyms] [k] get_page_from_freelist Namenode Java samples: Linux perf results:
  • 26. More about Linaro Connect: www.linaro.org/connect/ More about Linaro: www.linaro.org/about/ More about Linaro engineering: www.linaro.org/engineering/ ASIA 2013 (LCA13)