Putting Compilers to Work

Editor's Notes

• #2: This talk is about compilers from a practical perspective, about why compilers are important, and how and when to invest in building compilers.
• #3: Software = greatest force powering technology over last 50 years
• #4: - Reducing how many servers you need- Producing higher yields for your trading algorithms- Increasing user engagement with your product - Can make your product better (an edge over your competitors) in ways like improve your product’s battery life
• #5: [Live demo]
• #7: Unlike the demo, of course, real compilers are more complex and multiple stages that are organized into what’s called a “compilation pipeline”. Here on this slide, for example, we have a diagram of the compilation pipeline used by HHVM. The demo showed what happens at the very end of the compilation pipeline. Pipelines can differ a fair amount when comparing different compilers.. You’ll almost always see ASTs early in the pipeline, and mid-to-lower level SSA-based IR later in the pipeline. Also, for engines that take the “virtual machine” approach, you’ll typically see v-ISA around the middle of the pipeline. Pipelines are structured based on your constraints, your goals, and what kind of optimizations you want to focus on.
• #8: Many new languages were created during this period as developers faced new challenges involved in building web sites and web applications.
• #9: Most of the engines were interpreters that were in the ballpark of 100x slower than C/C++, but for small programs that mostly blocked on I/O the performance was acceptable.
• #11: Over the last 10-15 years, there were a lot of new efficient engines that came out for the popular dynamic languages: Javascript, PHP, Python, and Ruby. These advancements changed what was possible with web development. Sites could be richer, have more dynamic content, and be more interactive while still loading quickly.
• #12: Before digging into compilers and runtimes specifically, it’s important to dig into performance optimization in general
• #14: The diagram on this slide shows an example of an application that alternates between doing computation on the CPU and blocking on I/O. The x-axis is wall time. Notice how even if you made the “CPU” portions shrink to zero, the most you can reduce wall time here is about 50% (a 2x performance improvement)
• #15: Using async I/O APIs and task switch between multiple independent tasks to keep the CPU busy and avoid blocking
• #16: Small programs completely fit in cache, giving the illusion that memory is faster than it actually is. Larger programs will miss in the cache sometimes, Cache misses are similar to blocking on I/O, accept at a lower level of granularity. Instead of blocking dozens or hundreds of times, each time for a few milli-seconds (this is what I/O does).. with memory the CPU will stall many millions of times, each time for a tenth of a micro-second (death of a million cuts) Reduce the number of allocations, reduce the size of common data structures, avoid pointer-chasing and needless indirections where possible, when two variables are frequently read at the same time put the variables next to each other in your data structures Profiling what actually causes cache misses can be tricky. If you look where you took a cache miss, you’ll be able to identify the “victim” (i.e. what piece of data got evicted from the cache) but it will take some investigation to find the cause or the “perpetrator” (what piece of code polluted the cache and caused the other data to get evicted)
• #17: Use profilers to look at callstacks where the most CPU time is being spent. Don’t operate solely on hunches – even if you’re right most of the time, you might waste time focusing on the wrong things. Another technique that works sometimes: Get your hands dirty and trace through machine code in the debugger, focus on hot functions and common operations.
• #21: Short amounts of time: less than 1 ms or 100 ms Longer running: greater than 1 second or 1 minute or more
• #24: Interpreters: Instead of executing your program by compiling all the way to machine code (or some target language that is then compiled down to machine code). Interpreters run programs using a technique called “interpreted execution” where the interpreter engine is effectively executing on behalf of the source program.
• #25: If you plan on doing optimizations at the bytecode level, consider preferring a register-based bytecode over stack-based Design your bytecode so that opcode handlers can get most of what they need directly from an instruction’s immediate arguments As you squeeze for more performance, dispatch loop can be written in assembly
• #26: Depending on your use case, transpilers can deliver better performance that interpreters Like interpreters, there’s no need to worry about the low-level nitty gritty details of generating raw machine code No need to perform certain kinds of low-level compiler optimizations
• #27: Often has lower ceiling for best possible long-term performance vs. the full custom approach.. Now, the same can be said for interpreters, but I wanted to specifically call this out for traspilers because.. There’s often a misconception that if you traspile to C/C++ you’re going to get performance on par with hand written C/C++ programs Primitive operations that the source language exposes (whose precise semantics are relied on by most large codebases written in the source language) often do not cleanly map onto the primitive operations exposed by the target language Transpiling to C/C++ effectively locks you in to AOT for most use cases
• #28: Remember that LLVM only handles the end of your compilation pipeline for you (low-level IR to machine code), but the rest of the compilation pipeline (from source code down to low-level IR) is still on you. Doesn’t provide automatic memory management for you (no GC). It was built originally for an AOT compiler for a static language where fast compilation was not the top goal. LLVM has been incorporated into some major JIT-compilation-based engines for languages like JavaScript and PHP, but often these engines still have their own JIT and LLVM is often used as a second or third gear that is only used to compile the hottest ~1% of the program.
• #29: Benefits over LLVM: it’s a little higher level and provides GC for you InvokeDynamic was a promising development that came out ~5 years ago -> JRuby takes advantage of InvokeDynamic
• #30: Meta-tracing and partial evaluation frameworks are the newest approach when building efficient compilers. New and therefore not as proven as other approaches, but I think it’s a really exciting space and I’ll be curious to see what happens in the coming years.
• #32: Can go off the rails if you don’t have the proper expertise (it can go off the rails even if you do have expertise).
• #33: Depending on the approach you take, you’ll need to optimize different parts of the system. Obviously, optimize the parts of the system you control, but if you’re using an existing backend/framework you’ll need to make sure you’re doing the right things to get the best perf out of that technology. Use jemalloc or tcmalloc instead of the default allocator implementations from the C/C++ runtimes. If you have a huge amount of generated code, take advantage of Linux’s “huge pages” feature.
• #35: DB query compilation will be an interesting space to watch. With the rise of in-memory DBs, disk is not the bottleneck anymore. I think we’ll see different database systems striving to deliver the highest quality SQL->machine code compilation (and when I say that I’m referring both to execution performance of the machine code, as well as fast compile times)