Yes, But

Editor's Notes

• #2: The first leg of the epic, hopefully transcontinental book tour kicked off in my own backyard, at the technical annex of the huge Powell’s Books.
• #3: Just in case you were at Ignite Portland that night, here’s a recreation of my talk.
• #4: As the name implies, the book will teach you how to introduce automation into your user interface testing efforts, without making a bunch of false promises about its capabilities.
• #5: I was going to give the presentation this provocative title, as a signal that I’d be revisiting various points of view in the book and looking at them from a different angle.
• #6: But this title conveys the idea much more succinctly. If I can convey an idea from the book, and then talk about the edge cases where it doesn’t apply, then this talk will have a little something both for people who have read the book and those who haven’t.
• #7: So, let’s get started. One of my big motivations for writing the book was my dissatisfaction with testing tools that are based on recording a bunch of mouse clicks and then replaying them.
• #8: Such tools often generate unreadable code like this. How do you even know where you’re supposed to add the tests?
• #9: The problem is that many of these tools capture the wrong things. They dutifully record exact screen coordinates, instead of what the user was trying to do (e.g., delete a document, zap an alien, etc.).
• #10: It’s impossible to capture the user’s intent when all we have are keyboard and mouse actions, right? That’s the prevailing wisdom. But then I saw a talk by Dr. Atif Memon, who pretty much knocked all of us out of our chairs.
• #11: Instead of capturing raw mouse actions, Dr. Memon proposes to capture relationships between actions—specifically, the fact that one behavior, such as saving a document, frequently follows another.
• #12: From a directed graph of actions following actions, Dr. Memon’s team used a random walk to generate test cases. He didn’t say how they chose which nodes to traverse, but I suspect they used a dash of probabilistic techniques, such as the Markov Chains used for Garkov here.
• #13: Garkov, by the way, uses a corpus of Garfield captions to generate random, surreal dialogue for the comic strip. Here’s another one from Josh Millard’s site; I couldn’t resist.
• #14: Dr. Memon’s technique is a work in progress. But that doesn’t mean you can’t apply probability to your own test cases on a much smaller scale. The technique you’re about to see was inspired by Sammy Larbi’s article above, but with completely different Ruby code.
• #15: Let’s say you want to generate random sequences of words, based on the way they occur naturally in this text. For example, the word “red” will always be followed by “ball,” but the word “a” could have “ball” or “red” after it.
• #16: First, split the document into words.
• #17: Next, build a table linking each word to the list of all words that directly follow it, and how frequently they follow it. For example, you can see that “a” has been followed by two words: “ball” twice, and “red” once.
• #18: After all the counting is done, build a list of possible successors for each word. Use repetition to signal that one word has followed another one multiple times. That way, when you pick a random word from the list, you’re more likely to get a common follower than a rare one.
• #19: Now you can generate as many random words as you like. Simply grab a word, print it, and then choose a random item from the list of words that are “allowed” to follow it. Repeat until you’ve generated a sufficiently absurd document.
• #20: Here’s what the technique looks like when you apply it to the first chapter of my book. There are lots of ways to modify this technique, like looking at sequences longer than just pairs, or by considering letters instead of words.
• #21: You can also apply it to a test log instead of a sentence. Here’s a real sequence of actions I captured from my computer, and a brand new test script generated from them. As you can see, you really need a longer capture log before these scripts get interesting.
• #22: Okay, on to viewpoint #2. In the book, I talk a little bit about test-driven development and behavior-driven development in the context of automation. But there are plenty of times when completely manual tests give you information that you use to shape a product.
• #23: For example, when you turn off this touch-screen instrument and shine a light at the right angle...
• #24: ...you can see fingerprints on the most frequently-used parts of the interface. This is a developer’s machine, and he was comfortable using the on-screen data entry “knob” in the corner. But the machines we showed to customers came back with almost no prints on it.
• #25: After getting a brief facelift, the data entry control received a warmer reception from our users’ fingertips.
• #26: Even though this test started as a lo-fi, non-automated, analog kind of technique, there’s no reason we can’t go back after the fact and make an automated version. The forum linked above gives a quick intro to capturing mouse and keyboard events using C on Windows platforms.
• #27: Of course, this is a talk on Ruby, so let’s apply the techniques in Ruby using the win32 gem. First, there are a handful of Windows functions to define. We could define them one by one, like this.
• #28: But the book gives a technique (you can download the source for free) for automatically assigning Ruby names to the Windows functions.
• #29: The technique uses this method of turning Windows-style “CamelCase” names into Ruby-style “snake_case” ones.
• #30: Finally, you’ll need to define a few common Windows constants, and a shorthand for invoking the C functions from Ruby.
• #31: Here’s the meat of the mouse hook. Windows will call into this Ruby function when there’s mouse activity, passing in enough information to reconstruct exactly what happened. In this case, you’re just going to record the screen coordinates whenever the user drags his finger/mouse.
• #32: Here’s the bit of glue that makes the callback magic happen.
• #33: And finally, here’s the main loop. After you assign the mouse hook, you just sit in a message loop until you’ve logged all the data you want.
• #34: The output is a series of space-delimited x/y pairs, one per line. Each point represents one place where the user has dragged the mouse.
• #35: What can we do with that list of coordinates? How about a heat map? The article linked above describes a very detailed technique; I’m going to give a much lighter one that fits the broad scope of this talk.
• #36: You’ll need ImageMagick and its Ruby wrapper, RMagick. Start with a screen-sized empty black background, and a small tile to represent the “finger” smudging the screen.
• #37: Draw a semi-transparent white circle into the finger tile to represent the fingertip.
• #38: Now, bias the low end of the finger towardd blue, and the high end toward pink. There are much cleverer techniques that can get you a rainbow of hot and cool colors, but this simple trick will work well enough for now.
• #39: Now, load the screenshot into the picture and dim it quite a bit.
• #40: Then for each place the user clicked or dragged, add the fingertip image in. The effect is cumulative; frequently-touched parts of the screen will be brighter.
• #41: Here’s what the result looks like after a very short interaction.
• #42: And with a pang of guilt for throwing this stuff together practically on the eve of the talk, here are links to all the code you’ve seen.
• #43: The book purposefully skirts around the subject of testing philosophy, since that subject is extremely well covered. But I don’t want to leave the impression that the topic is completely exhausted.
• #44: Writers, coders, and testers are still challenging old notions and documenting new techniques. So please take tonight’s talk as a call to tinker, explore, and flip old approaches onto their heads.