Node.js Community Benchmarking WG update

Node.js Community
Benchmarking Efforts
Michael Dawson, IBM

About Michael Dawson
Loves the web and building software (with Node.js!)
Senior Software Developer @ IBM
IBM Runtime Technologies Node.js Technical Lead
Node.js collaborator and CTC member
Active in LTS, build, benchmarking , api
and post-mortem working groups
Contact me:
michael_dawson@ca.ibm.com
Twitter: @mhdawson1
https://www.linkedin.com/in/michael-dawson-6051282

4
About The Benchmarking Team
Love benchmarks and making things go faster.

5
Introduction to benchmarking
Node benchmarking workgroup
Real-life example
Agenda

Performance testing
Functional testing
6
Introduction
=

Fundamental run-to-run variance
– False positives
– Collecting enough samples to be sure of the result
– Documenting expected variance
Consistent environment
– Known starting machine state
– Machine isolation
– Interleave comparison runs
Jumping to conclusions
7
Introduction – Key Challenges

Change one thing and one thing only between runs
–Application code
–NPM module
–Runtime
–Machine
8
Introduction – Limit Entropy

9
Micro-Benchmarks
–Ex: Buffer.new()
Whole System
–Ex: AcmeAir - http://github.com/acmeair/acmeair-nodejs
Introduction – Approaches

I’ve found a regression in
…. now what ?
10
Introduction - Regressions

11
Introduction - Regressions
Are you sure ?
– Revalidate environment,
– Expected variance
If so, what changed ?
– Your application
– Node.js
– Your environment
Compare between good/bad cases
– Tools
– Binary search

12
Introduction – Sources of Regressions
Node.js
 lib/*.js – buffer, cluster, etc
V8
OpenSSL
libuv
NPM Module

13
Introduction – Tools
Javascript profiler
 V8 profiler
 Appmetrics - https://www.npmjs.com/package/appmetrics
 --trace-opt --trace-deopt
Native profiler (ex perf, tprof, oprofile)

14
Benchmark Workgroup
 Key goals
– Avoid Regressions
– Track and Evangelize gains between
releases
 How ?
 Define Use Cases
 Identify/Build Benchmarks
 Run/Capture results
https://github.com/nodejs/benchmarking

15
Benchmark Workgroup – Define Use Cases
https://github.com/nodejs/benchmarking/blob/master/docs/use_cases.md

16
Benchmark Workgroup – Runtime Attributes
https://github.com/nodejs/benchmarking/blob/master/docs/runtime_attributes.md

17
Benchmark Workgroup – Build/Identify Benchmarks
https://github.com/nodejs/benchmarking/blob/master/docs/case_coverage.md

18
Benchmark Workgroup – Build/Identify Benchmarks
https://github.com/nodejs/benchmarking/blob/master/docs/case_coverage.md

19
Benchmark Workgroup – Run/Capture Results
https://ci.nodejs.org/view/Node.js%20benchmark/

20
Benchmark Workgroup – Run/Capture Results
https://benchmarking.nodejs.org/

21
Benchmark Workgroup – Current Work
Expand benchmarks
Job -> run core benchmarks
 https://github.com/nodejs/benchmarking/pull/58

22
Example – The Code
var harness = require('../../common/harness.js');
var ARRAY = [1, 2, 23829, 4, 5, 7, 12312321, 2131, 434832, 43792, 23421, 65345, 132210,
77777, 322131, 1, 2, 23829, 4, 5, 7, 12312321, 2131, 434832, 43792, 23421, 65345, 132210,
77777, 322131, 1, 2, 23829, 4, 5, 7, 12312321, 2131, 434832, 43792, 23421, 65345, 132210,
77777, 322131, 1, 2, 23829, 4, 5, 7, 12312321, 2131, 434832, 43792, 23421, 65345, 132210,
77777, 322131];
var ITERATIONS = 300000;
var result;
function test() {
for(var i=0;i<ITERATIONS;i++) {
result = new Buffer(ARRAY);
}
}
harness.run_test(test);

23
Example – The Problem
Node 4.3.2:
./node benchmark.js
total time:5.079s / iterations:54 / ops/sec:10.63 / average time:0.09s /
variance:0.89%
variance:0.75%
Node 4.4.0:
./node benchmark.js
variance: 2.32%
variance: 0.28%
= ~ 40% regression

24
Example – V8 Profiler
 Part of Node.js binary
 Turn on with
– --prof
 Test-tick-process to post-process
– ./node --prof-process isolate-0x2818130-v8.log
 Other helper modules like
– https://www.npmjs.com/package/v8-profiler

25
Example – V8 Profiler
< 5585 23.7% 23.9% LazyCompile: *fromObject buffer.js:121:20
< 1308 5.6% 5.6% LazyCompile: *subarray native typedarray.js:165:28
< 1263 5.4% 5.4% LazyCompile: *Uint8ArrayConstructByArrayBuffer native
typedarray.js:35:42
< 964 4.1% 4.1% Builtin: JSConstructStubGeneric
< 854 3.6% 3.7% Stub: InstanceofStub
< 677 2.9% 2.9% LazyCompile: *test benchmark.js:7:14
< 669 2.8% 2.9% LazyCompile: *Uint8Array native typedarray.js:122:31
---
> 15227 47.1% 47.3% LazyCompile: *fromObject buffer.js:121:20
> 1240 3.8% 3.9% LazyCompile: *subarray native typedarray.js:165:28
> 1166 3.6% 3.6% LazyCompile: *Uint8ArrayConstructByArrayBuffer native
typedarray.js:35:42
> 967 3.0% 3.0% Builtin: JSConstructStubGeneric
> 802 2.5% 2.5% Stub: InstanceofStub
> 780 2.4% 2.4% LazyCompile: *test benchmark.js:7:14
> 654 2.0% 2.0% LazyCompile: *Uint8Array native typedarray.js:122:31

26
Example – Perf
perf record -i -g -e cycles:u -- ./node --perf-basic-prof benchmark.js
perf report
diff perf_good.out perf_bad.out
327c302
< 91.52% 23.43% node perf-16993.map [.]
LazyCompile:*fromObject buffer.js:121
---
> 93.25% 46.56% node perf-16934.map [.]
LazyCompile:*fromObject buffer.js:121
331c306

--trace-opt --trace-deopt
[marking 0x39570dd44951 <JS Function fromObject (SharedFunctionInfo 0x39570dd12f91)> for
recompilation, reason: not much type info but very hot, ICs with typeinfo: 14/64 (21%), generic ICs: 0/64
(0%)]
[compiling method 0x39570dd44951 <JS Function fromObject (SharedFunctionInfo
0x39570dd12f91)> using Crankshaft]
[optimizing 0x39570dd44951 <JS Function fromObject (SharedFunctionInfo 0x39570dd12f91)> - took
0.315, 1.339, 0.511 ms]
[completed optimizing 0x39570dd44951 <JS Function fromObject (SharedFunctionInfo
0x39570dd12f91)>]
[deoptimizing (DEOPT eager): begin 0x39570dd44951 <JS Function fromObject (SharedFunctionInfo
0x39570dd12f91)>
[deoptimizing (eager): end 0x39570dd44951 <JS Function fromObject (SharedFunctionInfo
0x39570dd12f91)>

28
Example – The Narrowing
Binary Chop
– Compare changes between good & bad
– git rev-list ^good bad^
– Potentially time consuming
– git bisect is quite useful

29
Example – The Result
• Issue in v8 optimiser.
• Will be fixed there once TurboFan
becomes default
• Until then…
• github.com/nodejs/node/pull/5819

Get Involved !
 http://www.github.com/nodejs/benchmarking
We need to expand coverage
Need help in tracking perf and investigating issues
28

Copyrights and Trademarks
© IBM Corporation 2016. All Rights Reserved
IBM, the IBM logo, ibm.com are trademarks or registered
trademarks of International Business Machines Corp.,
registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the Web at
“Copyright and trademark information” at
www.ibm.com/legal/copytrade.shtml
Node.js is an official trademark of Joyent. IBM SDK for Node.js is not formally related to
or endorsed by the official Joyent Node.js open source or commercial project.
Java, JavaScript and all Java-based trademarks and logos are trademarks or registered
trademarks of Oracle and/or its affiliates.
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or
both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft
Corporation in the United States, other countries, or both.
npm is a trademark of npm, Inc.

Node.js Community Benchmarking WG update

More Related Content

Viewers also liked (8)

Similar to Node.js Community Benchmarking WG update (20)

More from Michael Dawson (18)

Recently uploaded (20)

Node.js Community Benchmarking WG update