Cobi: A Community-Informed Conference Scheduling Tool. UIST 2013 slides
Download as PPTX, PDF2 likes837 views
Cobi is a community-informed conference scheduling tool designed to address the challenges faced by conference organizers, such as paper affinities and conflicting session preferences. It involves community sourcing to gather input from authors and committee members to create coherent sessions while resolving conflicts effectively. The tool enhances the scheduling process through mixed-initiative interfaces that allow organizers to make informed decisions based on community data and preferences.
Cobi: A Community-Informed Conference Scheduling Tool. UIST 2013 slides
- 1. Cobi: A Community-Informed Conference Scheduling Tool Juho Kim Paul André Wendy Mackay Robert C. Miller Haoqi Zhang Lydia B. Chilton Michel Beaudouin-Lafon Steven P. Dow 1
- 2. 420 papers 16 parallel rooms 200 sessions in 15 venues 4 days and 3442 attendees 2
- 3. Challenges for conference organizers 1. Lack data on paper affinities 2. Lack data on what people want 3. Lack software for resolving conflicts 101 102 103 104 4. Multi-dimensional requirements 1pm ? ? Room size Awarded papers ? good or bad? Session length ? Themes of the day 3pm Similar sessions in nearby rooms … 3
- 6. The Cobi Scheduling Tool 6
- 7. Conflicts how are they detected? how are they resolved? 7
- 8. Authors, do you want to see the other paper? Wikipedia Classroom Experiment A Pilot Study of Using Crowds in the Classroom ✓ ✓ ✓ X input mediation constraint ✓ Encode if two or more people express interest in seeing both. “papers of mutual interest should not be in opposing sessions.” 8
- 9. constraint conflict 1pm “Wikipedia Classroom Experiment” and “A Pilot Study of Using Crowds in the Classroom” should not be in opposing sessions. Room 1 Room 2 Wikipedia Classroom Experiment A Pilot Study of Using Crowds in the Classroom 9
- 10. Conflicts ✓ how are they detected? how are they resolved? 10
- 11. The Cobi Scheduling Tool 11
- 14. Step 1. Locate a conflict to resolve 14
- 15. Step 2. Study the conflict & propose move Conflict information 15
- 16. Step 3. Review recommendations and pick 16
- 17. Step 4. Make the final scheduling decision Showing how our conflict will be resolved by this move 17
- 19. Cobi Deployment for CHI 2013 19
- 21. 1 Communitysourcing Committee members create initial affinity scores between papers. 21
- 22. 22
- 23. Committee Clustering 1722 paper affinities from 64 committee members • Paul André, Haoqi Zhang, Juho Kim, Lydia B. Chilton, Steven P. Dow, and Robert C. Miller. Community clustering: Leveraging an academic crowd to form coherent conference sessions. HCOMP 2013, to appear. paper D ? paper B paper A ? ? paper C paper E ? 23
- 24. 1 Communitysourcing Authors judge relevance and interest. 24
- 25. 25
- 26. Authorsourcing 8651 preferences and constraints from 645 authors, covering 87% of accepted submissions my paper fits with my paper paper A paper D paper E does not fit paper B paper C paper Z 26
- 27. The value of community input “Authors were asked for input, most gave it, we tried hard to accommodate them, and almost nobody complained.” 27
- 28. 2 Encoding constraints Data from committee & authors Preliminary schedule from TP meeting Cobi detected 238 conflicts 28
- 30. Three organizers (general chair + TP chairs) 815 scheduling operations over 42 days 30
- 31. Conflicts: 238 70 (168 resolved) Conflict Count 250 conflict count 200 150 100 50 0 2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13 Date 31
- 32. Edit Count Conflict Count 180 250 session moves paper moves chair moves meta edits conflict count 150 120 200 150 90 100 60 50 30 0 2/10/13 0 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13 Date 32
- 33. Scheduling phases Resolving conflicts Themes & fine-tuning Session chair assignment Edit Count Conflict Count 180 250 session moves paper moves chair moves meta edits conflict count 150 120 200 150 90 100 60 50 30 0 2/10/13 0 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13 Date 33
- 34. Phase 1. Resolving conflicts Edit Count Conflict Count 180 250 session moves paper moves chair moves meta edits conflict count 150 Resolve author conflicts 120 200 150 90 100 Make coherent sessions 60 30 50 Adjust session length 0 2/10/13 0 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13 Date 34
- 35. Author conflicts All co-authors (not just presenters) can attend all of their paper presentations. 30 0 in 29 minutes 35
- 36. “Cobi trivialized conflict resolution.” 36
- 37. Phase 2. Themes and fine-tuning Edit Count Conflict Count 180 150 Switch rooms 120 250 session moves paper moves chair moves meta edits conflict count 200 Edit session titles 150 90 Reorder papers 100 60 Balance awards 50 30 0 2/10/13 0 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13 Date 37
- 38. Mixed-initiative problem solving “I was by and large driven by what Cobi was suggesting. As you make progress you can progressively integrate other criteria that are not explicit in the system.” 38
- 39. Phase 3. Session chair assignment Edit Count Conflict Count 180 250 session moves paper moves chair moves meta edits conflict count 150 120 200 150 90 100 60 Assign session chairs 50 30 0 2/10/13 0 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13 Date 39
- 40. http://projectcobi.com Frenzy session making committeesourcing Cobi scheduling tool authorsourcing myCHI for attendees 40
- 41. Paul André CMU Anant Bhardwaj MIT Lydia Chilton UW Juho Kim MIT Steven Dow CMU David Karger MIT Rob Miller MIT Haoqi Zhang Northwestern 41
- 42. Michel Beaudouin-Lafon Wendy Mackay Patrick Baudisch Upcoming 42
- 43. Cobi is a community-informed mixed-initiative system for a large-scale planning problem. Communitysourcing applications Encoding constraints from community data Mixed-initiative interface for conflict resolution 43
- 44. Cobi: Community-Informed Conference Scheduling Juho Kim | juhokim@mit.edu http://projectcobi.com/ 44
- 45. 45
- 46. Backup Slides 46
- 47. Different crowd, Different expertise • Organizers: 3 – Session balance, theme of the day • Committee members: 200+ – Affinities between papers in their subfield • Authors: 1000+ – Fit with their own paper • Attendees: 3000+ – Individual preferences for attending talks 50
- 48. Techniques for Affinity Creation • Manual grouping from TP meeting • Automatically generated affinity scores using TF-IDF • Committee-generated affinity scores using community clustering 51
- 49. Fine-grained affinity data • How relevant is this paper to yours? – “should it be in the same session as your paper?” • Is this paper interesting to you? – “Would you like to see this paper’s presentation?” 52
- 50. Session-related Constraint Type author with papers in opposing sessions topics of interest to a persona in opposing sessions Data Source Severity Encoded systemgenerated high systemgenerated medium - 53
- 51. Paper-related Constraint Type papers of mutual interests in opposing sessions papers that do not fit well in the same session Data Source Severity Encoded authorsourcing Preference Type papers good in the same session Data Source Severity Total authorsourcing N/A 805 high 923 authorsourcing medium 651 54
- 52. Chair-related Constraint Type chair's paper in own session chair's paper in opposing sessions chair interested in opposing sessions chair in a session with a bad fit Preference Type chair fits well in the session Data Source Severity Encoded systemgenerated high systemgenerated high authorsourcing medium chairs medium 243 - Data Source Severity Total chairs N/A 55
- 53. Constraint Type author with papers in opposing sessions topics of interest to a persona in opposing sessions papers of mutual interests in opposing sessions papers that do not fit well in the same session Related to Session Data Source systemgenerated systemgenerated Paper authorsourcing Paper authorsourcing medium systemgenerated high systemgenerated high authorsourcing medium chairs medium Session chair's paper in own session Chair chair's paper in opposing sessions chair interested in opposing sessions chair in a session with a bad fit Total violated Chair Chair Chair Preference Type papers good in the same session chair fits well in the session Total satisfied Paper Chair Severity Total Initial Final Change high - 31 0 -31 medium - 6 4 -2 923 40 19 -21 651 129 42 -87 - 21 0 -21 243 - 6 5 0 238 0 4 1 70 -6 -1 1 -168 high Data Source Severity Total Initial Final Change authorsourcing N/A 805 268 272 4 chairs N/A 90 78 -12 358 350 -8 56
- 54. System-defined conflicts over time Conflict Count author in opposing sessions persona in opposing sessions 30 Resolve author conflicts 20 Adjust session length Balance awards 10 Switch rooms 0 0 200 400 600 800 Edit Number 57
- 55. Community-defined conflicts over time papers that do not fit well in the same session papers of mutual interests in opposing sessions Conflict Count 120 Make coherent sessions Adjust session length Balance awards 90 60 30 Switch rooms 0 0 200 400 Edit Number 600 800 58
- 56. chair's paper in own session chair's paper in opposing sessions chair interested in opposing sessions chair in a session with a bad fit Conflict Count 30 20 Assign session chairs 10 0 270 420 570 720 Edit Number 59
- 57. 60
- 58. 61
- 63. Scratch space for unscheduled items 66
- 64. Scratch space for unscheduled items 67
- 67. Step 1. Locate a conflict to resolve 70
- 68. 71
- 70. 73
- 71. 74
- 72. 75
- 73. 76
- 74. 77
- 75. 78
- 76. 79
- 77. Consequence of a swap 80
- 78. 81
Editor's Notes
- #3: Walking around the room with paper index cards to form sessions. Paper-based, manual, dependent on the people in the room.Organizers spend months to refine the program, to resolve conflicts and take into account dozens of other factors.
- #4: Paper affinities, which tell us if two papers have a good fit in a sessionIt’s hard to know what people want in advance: some rooms get over-populated while others don’tConsequences, manually checkingMore than just conflicts – purely automatic methods cannot take into account all the subjective, implicit requirements.
- #6: Collecting rich data from community members, asking people to tell us papers have a good fit, and what they want to see.Encoding community data in a way that the system can handleScheduling tool supports conflict resolution
- #7: Segway to conflicts – it’s about conflict resolution.
- #17: Now the system enters the move mode,Which displays the session being moved in the notification panel and highlights in yellow.The consequences of swapping this session with other sessions are shown in numbers.The lower the better, indicating that more conflicts will be resolved.The recommended edits that can resolve most conflicts are highlighted in green.Now let’s review one of the recommended options to see if swapping is a good idea.
- #18: Now the system enters the move mode,Which displays the session being moved in the notification panel and highlights in yellow.The consequences of swapping this session with other sessions are shown in numbers.The lower the better, indicating that more conflicts will be resolved.The recommended edits that can resolve most conflicts are highlighted in green.Now let’s review one of the recommended options to see if swapping is a good idea.
- #21: Collecting rich data from community members, asking people to tell us papers have a good fit, and what they want to see.Encoding community data in a way that the system can handleScheduling tool supports conflict resolution
- #22: Collecting rich data from community members, asking people to tell us papers have a good fit, and what they want to see.Encoding community data in a way that the system can handleScheduling tool supports conflict resolution
- #25: Collecting rich data from community members, asking people to tell us papers have a good fit, and what they want to see.Encoding community data in a way that the system can handleScheduling tool supports conflict resolution
- #29: Collecting rich data from community members, asking people to tell us papers have a good fit, and what they want to see.Encoding community data in a way that the system can handleScheduling tool supports conflict resolution
- #30: Collecting rich data from community members, asking people to tell us papers have a good fit, and what they want to see.Encoding community data in a way that the system can handleScheduling tool supports conflict resolution
- #32: Note that not all conflicts are removed. Mixed-initiative. Subjective requirements beyond conflicts.
- #36: If Steven had 10 papers, he’ll be able to go to all ten of his paper presentations even if he’s not presenting them.
- #60: Note that it doesn’t start from 0.