CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak

Editor's Notes

• #2: Test.
• #3: Agenda. Walk through briefly.
• #4: Go!
• #5: DTrace was designed and implemented by Bryan Cantrill, Mike Shapiro, and Adam Leventhal. It was released in 2005. Since then it was open-sourced and for now it is supported by Solaris, Mac OS X, FreeBSD, NetBSD, Linux kernel (partially). Particularly OS X included dtrace in 2007 (version 10.5. Leopard) as part of Instruments testing suite. Stability was core assumption, that’s why there is no overhead when probes are disabled and also that’s why it’s limited in functionality.
• #6: DTrace mascot.
• #7: PROVIDER gives us general funcionality; MODULE sets the module we're focusing on (e.g. specific dylib); FUNCTION specifies function within module (this poses a limitation i.e. you can’t trace binaries that have their symbols stripped; ofc that also applies to unusal ‘calls’ like JMPing into code chunk instead of calling it – these will be invisible to dtrace); NAME gives us some idea about semantic meaning (e.g. entry/return, BEGIN/END, also when tracing a function you can specify offset within a function (at this offset you can e.g. peek into memory pointed by some register) or leave NAME blank to trace all the instruction within); PREDICATE acts as a conditional, and ACTIONS are what's gonna happen when the probe fires. NOTE: For PROVIDER:MODULE:FUNCTION:NAME you can use wildcards, e.g. *open* will trace any function with ‘open’ string in it. You can use Dtrace rapidly as one-liners and for more challenging tasks we can switch to scripting. Also, D scripts can be embedded into e.g. bash scripts to gain additional possibilities like argument parsing. Note: dtrace requires root privileges (interaction with kernel mode + destructive actions)
• #8: Example of dtrace script and a one-liner. Talk about dtrace provider and its BEGIN and END probe (they fire on starting and ending of a dtrace script).
• #9: Dtrace command invokes the compiler for the D language that outputs D Intermedaite Format which is sent to kernel part. As previously mentioned, Dtrace is pretty strict about corectness and guarantees safety with no additional overhead when probes are disabled (and in fact a system with disabled probes is identical to a system without dtrace at all). There is a possibility of stand-alone consumer as e.g. Pythons bindings. Dtrace providers are kernel modules that talk with dtrace kernel module through API.
• #10: You yourself can put static probes inside of the application, re-compile it and improve analysis with dynamic activation of the USDT probes when required (neglibile overhead when disabled). Possibilities examples: The JavaScript provider uses USDT to instrument the Mozilla JavaScript engine (Spider Monkey). It provides probes for function calls, object creation, garbage collection, and code execution. Basically you can use this provider to trace the operation of JavaScript code. I did not test it, so I’m not sure if this is still ‘the thing’ but the sole possibility is enough (e.g. woudln’t V8 equivalent be awesome? maybe)
• #11: D is a C-like language as they will see in a second
• #12: C-like syntax and functionality. Ptr dereferencing & traversing structure chains. Also, character escapes sequences are same as in C (e.g. \n = backslash n for newline)
• #13: We do not declare data type of a variable explicitly; Associative arrays = keys are tuples; We can declare a variable without initilization; When we zero-out variable it’s freed. Talk about global / clause / thread locality (when would we use it? Mention later examples) External variables are a way to access kernel variables in your Dtrace script. You need to pre-pend variable name with a backtick character to access them. Also, worth mentioning that Dtrace supports structs and unions along with typedefs. And even bit fields!
• #14: Talk about each and every one. Curpsinfo points to psinfo_t struct, curlwpsinfo points to lwpsinfo_t, and curthread points to kthread_t both are internal structures for the current process and thread. Give examples for args usage e.g. File descriptors from your scripts + Note: Args for C++ methods can be tricky to access, this is because it’s up to C++ compiler to organize arguments and you need to know how they’re organized before tracing operation (i.e. Which argument is the this ptr and are there any compiler bonus args). Ppid is parent PID. Double note that there is some more built-ins, worth checking it for yourself.
• #15: All this are C-like. Nothing much to say in this slide. Run through them and say e.g. ‘usual + - * /’ etc
• #16: This is due to guaranteed safety. Loops and ifs too easily can lead to never ending story (=break the system) which would break core assumption about safe usage on production systems. However, there are reserved keyword for loops, ifs, gotos, et cetera but they never saw an actual implementation (or a release, who knows what Sun/Oracle and later Joyent did).
• #17: Stack() / ustack() – self explanatory, display kernel and user mode stack. Tracemem() – dumps memory into the screen (peek a boo!) Alloca() – dynamic mem allocation inside of the dtrace script Bcopy() – might be used to copy data into newly allocated buffer Copyin() / copyinstr() / copyinto() – used to peek into data from user-mode processes (e.g. playing with user-mode requires usage of these in order to transer data to kernel) Msgsize() / strlen() – sometimes we want to measure sth
• #18: Talk for a second about pragmas, like quiet or destructive. Stop() – stopping process at point XYZ Raise() – sending signal, similar to kill cmd Copyout() / copyoutstr() – allows data modification, nemo used it to tamper function call arguments (he did so for x86 where fcalls args are usually all passed via stack; for x64 where first 6 args are passed via registers it might work if the argument is a pointer not a value => take the arg from register and mangle memory pointed by it) System() – execs an application
• #19: Breakpoint() – puts kernel-mode breakpoint (sucks if you don’t have connected debugger) Panic() – induces kernel panic at specific point Chill() – causing dtrace to spin for N nanoseconds, this is interesting when testing race conditions (you can slow down execution on purpose just to win races more often)
• #20: There is more providers and some of them might be interesting to _you_, e.g. Tcp/ip/udp providers might be interesting to sysadmins/network operators. Syscall – for tracing syscalls Pid – for tracing specific processes Objc – apple specific, more in ‘man dtrace’, for tracing specific objd functionality Fbt – function boundary tracing, you can use it to trace function from kernel (usage example for vulnerability analysis is in the ‘guide to kernel exploitation’ book) Proc – process creation and termination monitoring (nicely used in recent ‘launchd’ blog post by wuntee) Also, mention PHP and PYTHON providers as interesting (e.g. You can watch internals of the python script not the python VM when using python provider) and JS provider with conjunction to previously mentioned Firefox’s USDT.
• #21: Examples of active providers probes list (with counting!). Pid provider is a-ok because by default nothing uses it (that’s why it shows 0). Fbt is huge because you can probe most of the kernel functions.
• #22: Tiller and David’s talk touches the aspect of vulnerability research (ease of analysis + HIDS + code coverage). They also introduce „RE:Trace framework” a mixture of DTrace and Ruby bindings however I was not able to find it in teh Internetz. Also, I’m not a jeweller so that’s a no-no for ruby bindings. Nemo’s talk is mostly about rootkit-like functionality implemented via Dtrace (e.g. hiding files). I’m not so sure if this is the best course of action for gaining persistance but he presents interesting examples, mostly tampering with syscalls. In general context there is shit load of resources about general Dtrace usage, I will list most interesting ones in the reference section.
• #23: Regarding SystemTap: there even exists compatibility between Dtrace and Systemtap when it comes to USDTs Regarding provided link: Detours is the reason behind „mov edi, edi” hot-patch point inside of Microsoft’s DLLs. So it does introduce slight overhead even when disabled (as opposed to dtrace).
• #24: Go!
• #26: We’re snooping into Preview
• #28: These are logs for Safari execution
• #29: Googling for ideas is stil work-in-progress at google.
• #32: Global arrays for flagging FDs and MMAPs
• #33: Marking opened FD and printing out logging information.
• #34: Marking mmaps. Hm, but where’s our tracemem at mmap()’s return?
• #35: What? Tracemem() at munmap()? Well, as oppsoed to read() we can’t peek into memory at mmap()’s return even though the pointer is already valid. I found out that dtrace can’t peek into memory that was not previously touched and it seems that this is the case here. This has some down sides (memory could be altered at this point) but it’s better than nothing and I actually successfuly used it when tracking input for some OS X applications.
• #36: Just to be nice, we’re freeing closed FDs. This concludes input tracking, for working examples go to my blog and read latest post (even though for read() it’s basically the same).
• #37: Heap layout analysis when you’re performing heap exploitation (e.g. Can you somehow influence the heap layout? How reliably? Often times you can tinker with the heap but you don’t always get 100% reliability, then it’s good to know how many times your object is in the range where you want it to be). Custom memory allocators are also very interesting, mainly because if you would snoop only into system API for memory allocs you wouldn’t get much meaning (=application makes its own pools) but if you study the mechanism of the allocator then you can insert probes at appropriate functions via pid provider and get meaningful information. For kernel memory allocation we would need to utilize FBT provider in order to snoop into BSD wrappers or just go straight into the dragons den and snoop into MACH internals.
• #38: Note that when returning arg1 holds our return value instead of original argument. For other allocation functions (valloc, calloc, realloc, reallocf) probes would look similar, hence no point in going through all of them however I did include them for the sake of completness.
• #42: Sidenote: what’s the difference between realloc() and reallocf()? When reallocf() fails it frees source buffer (this call is FreeBSD specific to which OS X is closely related).
• #43: Freeing is as simple as it gets (not really, but for current version this is how we do things).
• #44: Output example. However we can pipe this into villoc and get visualizations!
• #45: Merging with villoc is an on-going project; we needed to discuss couple of things (e.g. Is memory allocation on OS X thread-safe or not? (Aparently it is, since it’s a POSIX requirement) and other things along what faults do we want to detect) in any way working alfa version is available on my github.
• #46: This is work in progress (mainly due to IDA side of the tool). It should be soon available on my github. Typical end goal for code coverage is to shrink an input pool for fuzzing operation of the application XYZ, I want to mark what code was touched for very specific input in order to speed up my analysis inside of a tool like IDA or Hopper. Yes, I am aware of IDA’s mac_servers for debugging integration. I’ve had some problems with them.
• #47: Regarding Python: When and if I finish python-based dtrace consumer I will open source it (if you’re interested you can follow me on twitter or github).
• #49: Questions and answers.