HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor

Editor's Notes

• #2: Thanks for inviting me. I’m a post-doc here at Stanford in the HCI group. I’ll be starting as Computer science faculty at Columbia in 2017.
• #3: I’m going to talk about humor but, I’m also going to give you some background on my area, which is Crowdsourcing, and HCI.
• #4: Let me quickly summarize everything I’m going to say in the next 90 slides.
• #5: The objective of Crowdsourcing is to decompose problems so people can collaboratively solve them. Crowdsourcing can be very simple tasks like image labeling or more complex tasks like Deciphering this message where no one person knows the answer. My work is on creative crowd algorithms that decompose harder and harder problems.
• #6: Other people in HCI have decomposed hard problems like design. In fact, didn’t have to use the crowd, there were able to automate some design problems. Maneesh Agrawala in the HCI group here has several papers that automatically create well-design Visual Communication: Generating readable map directions “how things work diagrams” Assembly instructions. All of these problems were decomposed in a similar way:
• #7: Humor is a difficult AI problem and very hard to decompose My work is inspired by Maneesh’s work, and treats humor generation as a constraint-based search.
• #9: Crowdsourcing was invented to use people for tasks that computer can’t do like labeling images.
• #10: Mechanical Turk is a platform where you can pay people a few cents to do a simple task. Turk quickly became very popular, But for a long time, Turk was only used to do embarrassingly parallel tasks like image labeling. So I asked: Can we do more complex things?
• #11: But my coauthor and I built a system called TurKit. which introduced the concept of Crowd Algorithms. You can write JavaScript programs that makes function calls to the crowd.
• #12: The most basic adaptive mechanism is improve-and-vote. Which allows workers to build on the work of others.
• #13: For example, Here is some messy handwriting. No one person could read it. I can’t even read it! Using improve and vote, the crowd could decipher it. Here’s how it works…
• #14: It starts with the image, At first we have no idea what it say, so we start with question marks Then we ask a worker to improve it, and they get parts of the answer, but other parts are wrong. Then we vote on which version is better. Usually the improved version is better, but not always. Sometimes you get spam, or just really weird interpretations. You take the version workers voted for, and you improve it again, until you get it all deciphered. This little portion says “You misspelled several words…” and goes on.
• #15: TurKit was widely adopted in research. Systems I built are in red, but other people used TurKit and built crowd-powered systems, too. Notably, Michael Bernstein at Stanford and Jeff Bigham at CMU built their first crowd systems in TurKit.
• #16: Cascade distributes big problems with complex interdependencies like creating taxonomies
• #17: Frenzy combines the knowledge of experts to meet a global constraint like Grouping papers into conference sessions.
• #19: Those patterns are all completed by humans, but if we understand the space well enough, it is possible to solve problems automatically with design principles. LineDrive is a system for creating useful driving directions based on visual principles.
• #20: But what people who give good directions do is more like this: How can a computer do that automatically? It’s so stylized? So different from a map. Methodology for automating the creation of these is to look at expert patterns.
• #21: Expert patterns, Observations from examples of expert work Grounded in psychological principles.
• #22: Design Space: angle you show the turn at. Can exaggerate it to emphasize the orientation of the turn. All Roads must be visible. Quantitative Evaluation: minimum road length (visibility threshhold), You want as many of the important roads as possible to meet this visibility threshhold.
• #24: Humor is a highly valued human skill. It is a sign of creativity and intelligence (Images – Monty Python. Obama laughin?)
• #25: Humor is used in many domains: - Persuasion in advertising and politics
• #26: Humor is used in many domains: - Persuasion in advertising and politics
• #27: It is used to capture attention
• #28: And it helps us learn. Because when we are having fun and at ease we learn better than when we are stressed and pressued.
• #29: Humor is hard. Humor is an AI Challenge Robots can’t do it. Why? Level of understanding experience and cretivity they don’t have. Data.
• #31: Incongruity theories the notion that humor violates our expectations. It is the detection of the incongruity that we find funny. Aristotle mentions this, modern philosophers such as Kant and Schopenhauer latched on It is now the dominant theory of humor.
• #32: Incongruity theories the notion that humor violates our expectations. It is the detection of the incongruity that we find funny. Aristotle mentions this, modern philosophers such as Kant and Schopenhauer latched on It is now the dominant theory of humor.
• #34: Humanity has been wondering about humor for over 2,000 years, so I’m obviously not go to start from scratch. My approach is to mine lots of examples and theories, and build a system that embodies them. It basically takes descriptive theories and turns them into generative theories.
• #35: For examples, I decided to narrow in on one very special form of humor. It’s made by The Onion, called American Voices. It takes real news headlines like this one: “People Bending iPhones At Apple Stores” And The Onion write fake man-on-the-street style responses: What’s special about this is that the headline and joke represent input and output pairs that we can study.
• #36: But what are we going to going to analyze these jokes for? Well, I distilled a lot of books on humor by philosophers, linguists, and comedians, and two ideas kept coming up: Associations Expectation Violation.
• #37: I prototyped a number of different generative models of humor, I settled on this model: Satisfying a constraint by search a space of associations. Expectation Violations is a goal. But it’s too hard to hit right off the bat. So I use associations to search for possible ways to violate expectations. By looking at examples, I found concrete mechanisms that could be applied. For example, Apple Store to Best Buy are Alternatives. That’s one type of association Best Best to Broken Displays is an Insult. That’s a different type of association. And there are multiple types of expectation violation. Let me explain each of those.
• #39: Like all good writing, details matter. Details bring the reader into the world your are writing.
• #40: Like all good writing, details matter. Details bring the reader into the world your are writing.
• #41: I prototyped a number of different generative models of humor, I settled on this model: Satisfying a constraint by search a space of associations. Expectation Violations is a goal. But it’s too hard to hit right off the bat. So I use associations to search for possible ways to violate expectations. By looking at examples, I found concrete mechanisms that could be applied. For example, Apple Store to Best Buy are Alternatives. That’s one type of association Best Best to Broken Displays is an Insult. That’s a different type of association. And there are multiple types of expectation violation. Let me explain each of those.
• #42: In the Best buy joke, it starts my aggressing with your expectation. This is horrible, I can’t believe it. Then at the last minute, it switches to be a Best Buy insult. I call this “bait-and-switch”.
• #43: For another example of Violation Mechanisms, here’s a different joke. The headline is HEALDINE The Onion writes: “So did he think Transcendence was good, or what?” You expect the focus to be the end for Mankind, and how horrible that would be. But instead, the focus is on the movie.
• #44: Lastly is sarcasm. Which hardly needs to be explained.
• #46: I prototyped a number of different generative models of humor, I settled on this model: Satisfying a constraint by search a space of associations. Expectation Violations is a goal. But it’s too hard to hit right off the bat. So I use associations to search for possible ways to violate expectations. By looking at examples, I found concrete mechanisms that could be applied. For example, Apple Store to Best Buy are Alternatives. That’s one type of association Best Best to Broken Displays is an Insult. That’s a different type of association. And there are multiple types of expectation violation. Let me explain each of those.
• #48: I built a system called HumorTools that teaches people the microtasks for creating humor and helps them follow them. The general guideline is that you Apply microtasks… Find a way of violating an expectation There are a lot of options, so just follow your train of thought until you get stuck, then backtrack.
• #49: Here is what one real user interface looks like: This asks users to decompose a headline into aspects. There are 20 of microtasks.. It’s too big to show you all of it. Instead, let me walk you through an example that illustrates the process.
• #50: We start with the headline Justin Bieber Basptized in NYC Bathtub
• #51: We break it into aspects
• #52: We pick the aspect “Justin Bieber” and start writing our expected reactions. I expect this is bad. His music sucks, he’s a bad role model.
• #53: But I can’t think of a violation mechanism for that.
• #54: So the I forget that chain of reasoning, and backtrack
• #55: Let’s try an association. Bieber has PR people who cover up for his stupid behavior. They’re the ones working hard to make it seem like he’s repentant.
• #56: Ah! And there’s a violation here! Although we expect the headline to be bad, From the perspective of the PR people it’s good
• #57: “Never let it be said that Bieber’s PR people aren’t brining new ideas to the table.”
• #58: And to get there we followed a train of thought, and did a little backtracking.
• #59: We can write multiple jokes this way.
• #60: If we start with Bathtub, we expect that to be dirty and gross
• #61: But to a fan, that they’d worship him anyway.
• #62: That’s a different unexpected angle joke.
• #64: I ran a study on HumorTools using 20 Stanford undergraduates.
• #65: I’m going to read you two headlines from the study. … The question is can users synthesize humor for them?
• #66: The evaluation showed that 75% of participatns were able 25% of the jokes were rated funny
• #67: Let me read show you some jokes: For the PETA headline where PETA seeks copyright for primate Here are two jokes for this headline: (read) Both have the look and feel of the Onion, and are amusing.
• #68: The first one was actually written by The Onion. The second was synthesized with HumorTools Raters found the HumorTools joke funnier than The Onion
• #69: For the Mars headline where Liquid Water Found on Mars, indicting the strong possibility of life on the red planet. Here are three jokes for this headline: (read) All 3 have the look and feel of the Onion, and are amusing.
• #70: The first two are humorTools And “sounds refreshing” is The Onion Both HumorTools jokes were funnier than The Onion
• #82: HumorTools is currently done completely by people, but since each microtask is fairly small, we could start to Train Machine learning systems with user-collected data, leading to hybrid human-computer joke generation
• #83: I have a grant to fund myself and 6 students this summer to be funnier than The Onion and release HumorTools