How to build a debuggle runtime

Editor's Notes

• #5: What is debugging? Debugging, in the most general sense, is finding out which assumptions are wrong about a system. Finding the wrong assumptions lead you a step closer to finding the problem Another way to look at it is: debugging is confirming all the things you believe to be true about a system.
• #6: If debugging is about finding which assumptions are wrong, what are the types of information that can help us. Historical Data – what happened? when did it happen? In which order did it happen? A common strategy is to put printfs in the code A similar technique is track events in logs Program State - what is the big picture view of the system at the time of the problem? Setting breakpoints to stop execution making it possible to inspect variables Generate a core dump to get a picture of the entire system at the time of the problem - Analysis – And once you have gathered all your data, you perform analysis to determine what the problem is or what additional data you need to find the problem
• #7: A debuggable runtime is a runtime that has: Builtin mechanism to capture historical data It always nice if logging Is a facility that you can simply turn on/off, instead of having to instrument code each time you run into a problem extract state of both the runtime and the language This is not trivial, external tools do not know enough about the runtime to distinguish between what is language state and what is runtime state Simplify or automate the process of analyzing the program state to find bugs Some bugs require a lot of effort to determine cause, if you have noticed patterns with these issues in the past you can add some checks to help you identify the problem in case you run into it again
• #8: So for this talk I will discuss What tools we have built into Openj9 to make debugging easier How did we create these tools What problems can we solve with these tools
• #9: Before we get into the JVM details I would like to first talk to you about Eclipse OMR – Open source Runtime toolkit Contains components that can be used to build language runtimes Components include GC – framework for managed heaps Compiler – components for building a compiler technologies (JIT compilers) Omrtrace – tracing library, also used for communication with IBM health centre monitoring tools Port library – platform porting library And more We have used some of these components in Openj9 to build debugging tools as you will see later
• #10: Constantly adding logging can be very burdensome and time consuming given that JVMs have many components many and 1000s of lines of code. A better solution is to add logging as you develop the JVM OMR TRACE component enables runtime developers to log internal and language events, which are called tracepoints. In openj9 they are used to log Gc activity, Jit compilation, classloading, etc as well as language events such as entering and exiting a method
• #11: Trace points are defined in a .tdf file. There is a .tdf for each component (one for gc, jit, vm, bcutil, etc. ) in the JVM There are different kinds of tracepoints each with a unique format (indentations for method entry/exit) Trace assertions are special in that they trigger system dumps (more on that later) Each trace point is associated with a level, so they can be turn on or off with commandline options
• #12: Tracepoints are stored in an internal circular buffer so only the most recent events are kept, and there is one for each thread Each tracepoint is timestamped and has a unique ID. Tracepoints can also be turned on indiviualling by specifying their ID They are stored in binary nonhuman readable format to conserve memory A tool called traceformat which comes with Openj9 is used to convert the binary data in to human readable format
• #13: Here is an example of this tool in action In this particular scenario we are dump all the trace out to the console (that’s the ‘:print’ part) And we are targeting all level2 and under tracepoints in the memory manager module In the example we see that we failed to allocate an object which triggers a local GC Notice timestamp and traceID
• #14: In this example we are tracing all methods in the j.l.Module class So we can trace but internal JVM events as well as language events Also it can show input parameters and return values This could be usefull for application developers to debug their applications as well
• #15: This is not a complete list but covers most of the things you can do with tracepoints Output: There are many configurations in which you can run the trace tool In the examples we saw how they can be outputted to the console. By default they are kept in internal trace buffers which can retrieved with a system core or using trace dump agents (more on that later) They can also be written to files or external listeners using the JVMTI RegisterTracePointSubscriber() (extended JVMTI feature) Quantity: You modify the trace buffer size to increase or decrease amount of history you want to keep track of There are also minimal and maximal modes which determine how much information each tracepoint stores (minimal only stores traceID and timestamp) Content We have tracpoints in all JVM components, interpreter, GC, JIT, Classloading, internal natives, etc Lets you trace java method entry/exit, also you can set it to show input parameters and return values Control: Trace points can be triggered to sleep, resumes, suspend if certain conditions are met. Ie. When method Foo is called sleep for 10seconds
• #16: Trace shows you the steps leading up to a problem. Ie. You have some corruption which make you jump to random location, tracepoints will help pinpoint where that corruption happened Usefull incases where a certain unpredictable order of events may cause a failure, Trace will help you see which order these events occurred Trace lets you see what parameters were passed in leading to the problem
• #17: Openj9 comes with dump agents that allow you to configure the conditions for when you want the different types of dumps to occur. This is done using the -Xdump JVM command line option. The types of dumps are, system, heap, java, trace, stacktraces, thread and jitdumps Stack dumps write basic information to stderr. System, java and heap give you a picture of the entire JVM state JIT is diagnostic data for the JIT compiler. Each dump agent is triggered by a configurable event, e.g. filtered exception or loading a specific class (it’s regex, so it could even be any class in a package). You can use specify multiple dump agents to cover different scenarios. For example, you can get a heap dump on every GC or on OOM, a system core if a GPF occurs, and a java core when the JVM shuts down. Xdump also allows you to preform requests such as a heap compaction dumping a system core
• #18: Openj9 has a hook mechanism built on omr hook gen This is used for events such as classloading, GC, startup/shutdown, etc. for jvmti agents, jit bookeeping (flush method cache if class is unloaded) Xdump piggybacks on the hook system and registers listeners for certain hooks depending on the chosen configuration When hooks are triggered dump agent checks the event data to see if it matches the filter (ie. did the thrown exception match the filter)
• #19: When we do get a match, al threads are halted For system dumps we use an omrport function to do the system dump For javacores and heapdumps we use a combination of jvmti heap iterators and vm classtable iterators to collect the necesarry data and write out to a file For trace dumps we simply write the trace buffers to a file
• #20: Here are some examples of the xdump option The first dumps a system core when a Null pointer exception is throw in TestClass Shows you can filter on certain typs of exceptions, in certain classes (or even methods within those classes)
• #21: The second produces a heap dump when it receives a user signal (kill -3 [PID])
• #22: And the third lists all the available options This is not a complete list, but entering that command will show you all the options
• #23: This very usefull for debugging hangs, you could configure it to generate a sequence of dumps to see whether any progress is being made It can also be used to debug intermittent issues, ie. Produces a core when a certain exception is thrown. And since you have the complete state, you can use it to find user level problems as well
• #24: DDR (dynamic dump reader) allows users inspect system cores It contains layout info of the JVM process, all the data structures, and can map out the entire JVM DDR is also aware of internal JVM algorithms. For example, it knows how to walk the Java stack. Or find the root path for a given object This tool Is also extensible, so developers can add their own custom commands With this knowledge, we get a powerful debugging tool… DDR-gen generates blob and java code (more on that later) required for ddr interactive
• #25: compile the JVM with debug symbols Scan source for macros and compiler flags Ddrgen which makes use of libdwarf to inspect the compiled library to find sizes and offsets and output to blob.dat. Structures can be compiled to different size on different platforms, void* will be different on 32 and 64bit platforms, same as a long. Some platforms will add padding and some will not. Blob.dat is loadd into memory at runtime superset.dat contains names of all structures and their members
• #26: Use the superset file to generate sturcture and pointer classes Functions have getters for all members of stuctures, also have gettters for address of members
• #27: When ddr is started it loads the generated classes and the ddr blob At this point we know all the structures and all their sizes and offsets This appproach also lets us be platform agnostic because all the platform information we need (sizes and offsets) is in the core
• #28: Here is how everything fits together At AOT we do the DDR generation (superset, generated classes, blob) At runtime JVM contains DDR blob When a system dump is triggered, the core also contains DDR blob DDR makes use of use of superset, generated classes and blob
• #29: Here is an example print all java threads and inspect java stacktrace It gives me the address of the Java thread, the thread ID and the name. But it also gives my suggestions for commands to further explore the threads. !j9vmthread will let me explore the J9 VM thread structure stored at that address. !stack will give me the information about the thread stack. This is possible because DDR knows how to walk a Java stack. Let’s run the !stack command on the main thread. This is just a test scenario, so we only get this minimal thread stack. We get another suggestion – we can explore the method(s) that are on the stack. … and I can keep digging: I can explore the constant pool or take a look at bytecodes of this method. This is particularly useful when looking at bytecodes that are modified when loaded, for example the way J9 handles MethodHandle invocations.
• #30: Here is different stack trace with more detail In this case, we see frame details like SP, BP, A0 We can also see the address of local variables We can also see whether the method is interpreted or Jitted which is usefull for problem determination
• #31: Here is an example where we dump a string object We can see all the fields, even the “hidden” ones like the class reference
• #32: And it doesn’t just print things out, it can also do some analysis for you We have a deadlock command that will show you any deadlocks. Requires no effort on your part which is nice
• #33: There are many more features, you can find them by !j9help
• #34: GDB – they can also inspect core files but it doesn’t know enough about the VM to print out structures (which will require debug symbols) or objects (which it simply cannot do). - Requires you to understand the JVM (where is the starting point, how do I make sense of these structures) DDR can explore the system core without source code and debug symbols. And, in the spirit of Java, it’s backwards compatible, so you can read core dumps produced by older JVMs because DDR keeps track of structure versions. Also GDB is platform specific whereas you can take a core from an AIX PPC 64-way machine and debug it on an intel ubuntu laptop using DDR
• #35: Just to drive home the point Here is a java stracktrace on GDB and DDR As you can see one is a lot easier to read and understand
• #36: Xcheck is a verification tool built into the JVM It can run checks on GCs, JNI calls and memory alloc/dealloc
• #37: Leverages hook mechanism for gc events A bascially performs sanity checks for GC bookeeping Any bad objects on the stack Are Remembered set slots actually pointing to different generations Is the slot in the object really a class pointer Us the heap walkable This checks make it very easy to find bugs that could take a long time to discover
• #38: Little example of the tool in action,
• #39: Basically your regular jni call with additional checks for Args types/ values Calss pointer Env point Exception checking Also does function tracing
• #40: Very usefull for JVM developers but JVM users could also make use of this to verify their JNI code Our service team has found many bugs user JNI code, and a feature like this makes it less likely
• #41: At startup we install new port lib with new alloc and free functions Allocators can add graurd pages around requested memory region Deallocators write over freed memory and do header and footer checks for corruption,(very usefull for detecting underflows and overflows) Guard pages can be mem protected so you can find out exactly when corruption occurred And we keep track of all allocations in a list and check at shutdown for all unfreed blocks We keep a memtag so we know file and line number of each allocation
• #42: Helps out a lot with analysis, Is the problem in the JVM? Is it in user code? Is it a GC problem? In the VM? Are we leaking memory? And its all automated!
• #44: You’ve seen what we have done in openj9, what can you do in your applications to make debugging easier