![Crowd Intelligence in Requirements Engineering: Current status and future directions
The Crowd-based Requirements Engineering Activities
RE Activities Perspectives/Activities No of
Studies
Elicitation
General Requirements [1,13,34,44,39,50,56,57,58,70,77,84,87] 70
Features [25,49,69,58,90,91]
NFRs [4,5,23,51,72]
Run Time Feedback [71,22,31,41,72,96]
Emerging requirements [15,42,57]
Design Rationale [37]
Modelling &
Specification
Use cases [27,46,], process models [12,26], Goals [61,68], i*[59], feature
models [19,68]
8
Analysis and
Validation
By crowd [48,54,72], by textual data analysis
[13,20,25,33,41,42,49,51,52,62,64,80,86,88,89], by prototyping [22],
sentiment analysis [21,79], image and unstructured data analysis [21,73]
22
Prioritization User rating and comments [14,43], developer voting [72], crowd members
vote [70,72], statistical analysis [21], gamified approaches [36]
6
Run-time Monitoring [2,20,73,75,93], adaption [3,26,55], evolution [25,45,66,85],
discovery [81], context [21,42,48]
16](https://image.slidesharecdn.com/crowdre-v2-190326081550/85/Crowd-Intelligence-in-Requirements-Engineering-Current-Status-and-Future-Directions-6-320.jpg)
![Crowd Intelligence in Requirements Engineering: Current status and future directions
Crowd mentioned in RE Literature
Attributes Explanations
Definition Crowd are the entities who will take part in the requirements process
Properties Mainly classified into three properties:
• Scale (usually deals with large crowd)
• Level of skills (required skills of the crowd in a specific subject domain)
• Roles ( remit and expectation of the crowd members involved in CrowdRE)
Literature • Techniques proposed by Srivastava and Sharma [72], Levy et al [39] and Groen
[22], where’s macro users communities were involved to elicit requirements.
• Munante et al [55] gather preferences of both domain experts and end users
configuration requirements for adaptive systems.
• Snijders [70], proposed a CrowdRE approach that gathered requirements are
analyzed and prioritized by involving crowd members.
• Groen et al [21] requirements gathered are validated by developers or third-party
experts.](https://image.slidesharecdn.com/crowdre-v2-190326081550/85/Crowd-Intelligence-in-Requirements-Engineering-Current-Status-and-Future-Directions-8-320.jpg)
![Crowd Intelligence in Requirements Engineering: Current status and future directions
Task assigned to crowd in RE
Attributes Explanations
Definition Task is the requirement activity in which the crowd participates.
Properties Task in CrowdRE are categorized according to:
• Type (refers to the nature of task for which crowd will participate)
• Extraction of raw requirements [34]
• Provide feedbacks[48,49]
• Bug identification [21,31,20,73]
• Feature request identification [29,52]
• Non functional requirements [31,72]
• Complexity (whether simple, medium or complex)
• Complexity is inter-related with crowd role, level of skills and time needed
by the crowd to perform it](https://image.slidesharecdn.com/crowdre-v2-190326081550/85/Crowd-Intelligence-in-Requirements-Engineering-Current-Status-and-Future-Directions-9-320.jpg)
![Crowd Intelligence in Requirements Engineering: Current status and future directions
Collaboration and Aggregation Mechanisms for Crowd RE
Attributes Explanations
Definition • Collaboration means that whether individuals in the crowd need to collaborate to
complete a task
• While aggregation means that individual’s contributions are aggregated to present
some useful information
Literature Groen et al [20, 21] proposed theoretical models for CrowdRE using concepts of
crowdsourcing where individuals’ tasks are aggregated to provide a final list of
identified requirements](https://image.slidesharecdn.com/crowdre-v2-190326081550/85/Crowd-Intelligence-in-Requirements-Engineering-Current-Status-and-Future-Directions-10-320.jpg)
![Crowd Intelligence in Requirements Engineering: Current status and future directions
Media/Channel
Type of Media Researchers used Media type
Twitter Guzman et al [22, 25], Williams & Mahmoud [79]
User Forums Bakiu et al [4] (epinions.com), Do et al [15] (Firefox, Lucene and Mylyn), Greenwood et
al [19], Kanchev et al [34, 35] (Reddit.com), Li et al [41] (sourceforge.net), Qi et al [62]
(Jd.com) , Xiao et al [80] (epinions.com), Shi et al [69] (JIRA)
LinkedIn Groen [22], Srivastava & Sharma [72]
Mobile stores Groen et al [23], Johann et al [33], Maalej and Nabil [49], Williams et al [87], Dhinakaran
et al [88], Liu et al [91]
Amazon store Groen et al [23], Kurtanovic and walid [37]
Facebook Hamidi et al [26]
Issue Tracking Merten et al [52] (GitHub ITS & Redmine ITS)
MTurk Murukannaiah et al [56, 57] , Breaux et al [8], Gemkow et al [89], Khan et al [90]
• In order to gain access to massive crowd, we need certain media.
• CrowdRE uses different channels to achieve this](https://image.slidesharecdn.com/crowdre-v2-190326081550/85/Crowd-Intelligence-in-Requirements-Engineering-Current-Status-and-Future-Directions-11-320.jpg)
![Crowd Intelligence in Requirements Engineering: Current status and future directions
Incentives/Motivation
Attributes Explanations
Definition • To engage the crowd in feedback generation or requirement elicitation, certain
motivation and incentives strategies are required
• Most common motivations are
• Rewards [1, 56]
• Gamification and public acknowledgement [3, 70, 73] or
• Multiple techniques in combination.
Literature • Snijders et al [71], propose REfine, a game-based requirements elicitation
technique which use gamification to constantly motivate the crowd members for
giving feedback and keeping them involved.
• Srivastava and Sharma [72], Levy et al [39], Groen [22] and Munante et al [55]
propose that rewards and acknowledgements can be used to motivate experts and
non-experts crowd members](https://image.slidesharecdn.com/crowdre-v2-190326081550/85/Crowd-Intelligence-in-Requirements-Engineering-Current-Status-and-Future-Directions-12-320.jpg)
![Crowd Intelligence in Requirements Engineering: Current status and future directions
Quality of collected data
• It is important to evaluate and ensure the quality of requirements obtained from
the crowd either as individuals or as groups.
• Unfortunately, this problem remains largely uninvestigated in CrowdRE literature.
• It is well argued in general crowd intelligence literature [94] that diverse,
independent and decentralized crowd performs better than experts in certain
circumstances.
• Researcher’s needs to explore this part further in future research, as up to date,
according to our knowledge there is less research study and needs further
exploration.](https://image.slidesharecdn.com/crowdre-v2-190326081550/85/Crowd-Intelligence-in-Requirements-Engineering-Current-Status-and-Future-Directions-13-320.jpg)
![Crowd Intelligence in Requirements Engineering: Current status and future directions
Foci of CrowdRE involving data analysis
• Maalej et al highlighted the issues and emphases to use automated techniques to
categorize users’ feedbacks into different categories in crowd-based requirements
engineering.
• Recently, Sarro et al [86], used Case Based Reasoning (CBR) algorithm to predict
mobile apps rating based on the features claimed for mobile apps in their description.
• Gemkow et al [89], applied AI techniques to a crowd-generated dataset, to extract the
domain-specific glossary terms.
• Seyff et al [85], propose to use AI techniques together with crowd-generated data in
order to observe the effects of requirements on sustainability
• Dhinakaran et al [88] proposed an active learning approach to classify requirements
into features, bugs, rating and user experience.
• Khan et al [90], propose semi-automated goal modeling approach to model features
identified from CrowdRE.
• Williams et al [87] proposed that automated social mining and domain modeling
techniques can be used to analyze mobile app success and failure stories to identify
end-users’ concerns of domain
• Oriol et al [93], proposed a framework to simultaneously collect feedbacks and
monitoring data from mobile and web users.](https://image.slidesharecdn.com/crowdre-v2-190326081550/85/Crowd-Intelligence-in-Requirements-Engineering-Current-Status-and-Future-Directions-14-320.jpg)
Crowd Intelligence in Requirements Engineering:Current Status and Future Directions
- 1. REFSQ’19, Essen, Germany Crowd Intelligence in Requirements Engineering: Current Status and Future Directions Javed Ali Khan1 Lin Liu1 Lijie Wen1 Raian Ali2 Presenter : Fabiano Dalpiaz 1Tsinghua University, School of Software 2Bournemouth University, Poole, United Kingdom
- 2. Crowd Intelligence in Requirements Engineering: Current status and future directions Motivation • Software Systems are engineered via interactive processes between multiple stakeholders. • Conventional RE approaches often rely on limited number of stakeholders. • But it is possible today to involve a large group of potential users and contributors distributed geographically and culturally. • Today RE can benefit from novel techniques and tools to support converging crowd intelligence in requirement elicitation and decision. • Our Observation: • “ To cater for the diversity of crowd in RE, more efforts have to be made to provide further support.”
- 3. Crowd Intelligence in Requirements Engineering: Current status and future directions Research Objectives and Questions • This is a literature review to evaluate the current status of the field of Intelligent Crowd-based RE. • RQ1. What are the current foci of CrowdRE research? • RQ2. How traditional RE activities are enhanced with Crowd activities, i.e. how Crowd-based techniques support RE activities? • RQ3. What is a possible future role of intelligence in CrowdRE? Then, we propose a general research map and the possible future roles of crowd intelligence in RE.
- 4. Crowd Intelligence in Requirements Engineering: Current status and future directions Research Method
- 5. Crowd Intelligence in Requirements Engineering: Current status and future directions Category of papers
- 6. Crowd Intelligence in Requirements Engineering: Current status and future directions The Crowd-based Requirements Engineering Activities RE Activities Perspectives/Activities No of Studies Elicitation General Requirements [1,13,34,44,39,50,56,57,58,70,77,84,87] 70 Features [25,49,69,58,90,91] NFRs [4,5,23,51,72] Run Time Feedback [71,22,31,41,72,96] Emerging requirements [15,42,57] Design Rationale [37] Modelling & Specification Use cases [27,46,], process models [12,26], Goals [61,68], i*[59], feature models [19,68] 8 Analysis and Validation By crowd [48,54,72], by textual data analysis [13,20,25,33,41,42,49,51,52,62,64,80,86,88,89], by prototyping [22], sentiment analysis [21,79], image and unstructured data analysis [21,73] 22 Prioritization User rating and comments [14,43], developer voting [72], crowd members vote [70,72], statistical analysis [21], gamified approaches [36] 6 Run-time Monitoring [2,20,73,75,93], adaption [3,26,55], evolution [25,45,66,85], discovery [81], context [21,42,48] 16
- 7. Crowd Intelligence in Requirements Engineering: Current status and future directions CrowdRE Essentials • We found that researchers focus was on the following aspects when it comes to crowd-based approaches • The Crowd – Who are they? • The tasks delegated to the crowd – What they do? • The design of mechanisms • Crowd are a competing or collaborating? • What is the media / channel for communication between crowd members? • What are the Incentives for engaging the crowd? • What are the ways to evaluate the quality of deliverables form the crowd?
- 8. Crowd Intelligence in Requirements Engineering: Current status and future directions Crowd mentioned in RE Literature Attributes Explanations Definition Crowd are the entities who will take part in the requirements process Properties Mainly classified into three properties: • Scale (usually deals with large crowd) • Level of skills (required skills of the crowd in a specific subject domain) • Roles ( remit and expectation of the crowd members involved in CrowdRE) Literature • Techniques proposed by Srivastava and Sharma [72], Levy et al [39] and Groen [22], where’s macro users communities were involved to elicit requirements. • Munante et al [55] gather preferences of both domain experts and end users configuration requirements for adaptive systems. • Snijders [70], proposed a CrowdRE approach that gathered requirements are analyzed and prioritized by involving crowd members. • Groen et al [21] requirements gathered are validated by developers or third-party experts.
- 9. Crowd Intelligence in Requirements Engineering: Current status and future directions Task assigned to crowd in RE Attributes Explanations Definition Task is the requirement activity in which the crowd participates. Properties Task in CrowdRE are categorized according to: • Type (refers to the nature of task for which crowd will participate) • Extraction of raw requirements [34] • Provide feedbacks[48,49] • Bug identification [21,31,20,73] • Feature request identification [29,52] • Non functional requirements [31,72] • Complexity (whether simple, medium or complex) • Complexity is inter-related with crowd role, level of skills and time needed by the crowd to perform it
- 10. Crowd Intelligence in Requirements Engineering: Current status and future directions Collaboration and Aggregation Mechanisms for Crowd RE Attributes Explanations Definition • Collaboration means that whether individuals in the crowd need to collaborate to complete a task • While aggregation means that individual’s contributions are aggregated to present some useful information Literature Groen et al [20, 21] proposed theoretical models for CrowdRE using concepts of crowdsourcing where individuals’ tasks are aggregated to provide a final list of identified requirements
- 11. Crowd Intelligence in Requirements Engineering: Current status and future directions Media/Channel Type of Media Researchers used Media type Twitter Guzman et al [22, 25], Williams & Mahmoud [79] User Forums Bakiu et al [4] (epinions.com), Do et al [15] (Firefox, Lucene and Mylyn), Greenwood et al [19], Kanchev et al [34, 35] (Reddit.com), Li et al [41] (sourceforge.net), Qi et al [62] (Jd.com) , Xiao et al [80] (epinions.com), Shi et al [69] (JIRA) LinkedIn Groen [22], Srivastava & Sharma [72] Mobile stores Groen et al [23], Johann et al [33], Maalej and Nabil [49], Williams et al [87], Dhinakaran et al [88], Liu et al [91] Amazon store Groen et al [23], Kurtanovic and walid [37] Facebook Hamidi et al [26] Issue Tracking Merten et al [52] (GitHub ITS & Redmine ITS) MTurk Murukannaiah et al [56, 57] , Breaux et al [8], Gemkow et al [89], Khan et al [90] • In order to gain access to massive crowd, we need certain media. • CrowdRE uses different channels to achieve this
- 12. Crowd Intelligence in Requirements Engineering: Current status and future directions Incentives/Motivation Attributes Explanations Definition • To engage the crowd in feedback generation or requirement elicitation, certain motivation and incentives strategies are required • Most common motivations are • Rewards [1, 56] • Gamification and public acknowledgement [3, 70, 73] or • Multiple techniques in combination. Literature • Snijders et al [71], propose REfine, a game-based requirements elicitation technique which use gamification to constantly motivate the crowd members for giving feedback and keeping them involved. • Srivastava and Sharma [72], Levy et al [39], Groen [22] and Munante et al [55] propose that rewards and acknowledgements can be used to motivate experts and non-experts crowd members
- 13. Crowd Intelligence in Requirements Engineering: Current status and future directions Quality of collected data • It is important to evaluate and ensure the quality of requirements obtained from the crowd either as individuals or as groups. • Unfortunately, this problem remains largely uninvestigated in CrowdRE literature. • It is well argued in general crowd intelligence literature [94] that diverse, independent and decentralized crowd performs better than experts in certain circumstances. • Researcher’s needs to explore this part further in future research, as up to date, according to our knowledge there is less research study and needs further exploration.
- 14. Crowd Intelligence in Requirements Engineering: Current status and future directions Foci of CrowdRE involving data analysis • Maalej et al highlighted the issues and emphases to use automated techniques to categorize users’ feedbacks into different categories in crowd-based requirements engineering. • Recently, Sarro et al [86], used Case Based Reasoning (CBR) algorithm to predict mobile apps rating based on the features claimed for mobile apps in their description. • Gemkow et al [89], applied AI techniques to a crowd-generated dataset, to extract the domain-specific glossary terms. • Seyff et al [85], propose to use AI techniques together with crowd-generated data in order to observe the effects of requirements on sustainability • Dhinakaran et al [88] proposed an active learning approach to classify requirements into features, bugs, rating and user experience. • Khan et al [90], propose semi-automated goal modeling approach to model features identified from CrowdRE. • Williams et al [87] proposed that automated social mining and domain modeling techniques can be used to analyze mobile app success and failure stories to identify end-users’ concerns of domain • Oriol et al [93], proposed a framework to simultaneously collect feedbacks and monitoring data from mobile and web users.
- 15. Crowd Intelligence in Requirements Engineering: Current status and future directions Research Map for Intelligent CrowdRE Crowd RE Activities Crowd Tasks Mechanisms Scale Expertis e Role Types Complexit y Collaboration/ aggregation Media/ channel Incentives/ Motivation Quality Elicitation Small - Large Low General system user Feature Requests, New requirements simple collaboration between crowd/ aggregation in final outcome Twitter, User forums, Facebook, Online websites, Mobile app stores Gamification Vouchers, Social recognition, Cash No individual guarantee, by statistical analysis Modeling Small - mediu m High Analysts & domain experts in dev. team Co-modelling, goal modeling, feature modeling, process modelling, argumentation complex Direct/indirect collaboration Platform-based Gamification, Assigned or obliged Relying on Individual expertise Analysis information retrieval, sentiment analysis, language patterns, recommender system No collaboration between crowd/ aggregation in final outcome Manual or automated text analysis, speech act recognition tools Gamification , Social recognition, cash, assigned or obliged Individual expertise Validation developer s team or 3rd party Annotation or walkthrough review Complex but could be partially or fully automated Validation need to check the influence Relying on Individual expertise Runtime adaptation/ evolution Large Medium- high Users behavioral Logs analyzed by developm ent team Feedbacks on Bugs, monitoring run time logs of exceptions, abnormal behaviors Log analysis and mining tools Users provide feedbacks voluntarily or obliged Fairly reliable Prioritization and Negotiation Small - Large Low General system user Preference elicitation, win- condition elicitation simple Direct/indirect group decision making Voting or group decision making tools Gamification Vouchers, Social recognition, Cash No individual guarantee, by consensus
- 16. Crowd Intelligence in Requirements Engineering: Current status and future directions Elicitation & Analysis in CrowdRE • Must work has been done on requirements elicitation using the crowd. • Requirements activities in CrowdRE are mostly focusing on user feedbacks. • Users may give feedback in the form of images, audio or video, thus analysis is required to deal with such unstructured data, which is still need to be explored. • AI techniques such as swarm algorithms, case-based learning, sentimental analysis, information retrieval algorithms, usage mining and collaborative filtering can be used with crowd-generated data to get useful insights.
- 17. Crowd Intelligence in Requirements Engineering: Current status and future directions Motivation in CrowdRE Crowd participation are of potential aid for all RE activities. • The added value by the crowd member is the sense of participation itself where the crowd feel relatedness and ownership of the solutions all the way through the development process. • Social principle can be adopted for motivating crowd members. • Fun and enjoyable activities like visual effects, animations can be used to motivate and engage crowd member. • AI intelligent approaches and analytics can play important role in predicting when incentives shall be given and in which format. • One-size-fits all style for motivation would not work and personalization and cultural-awareness are needed
- 18. Crowd Intelligence in Requirements Engineering: Current status and future directions Modelling & Validation in CrowdRE • Quite few research work have been identified till date on CrowdRE modeling and validation. • We suggest direct or indirect collaboration support to incorporate crowd intelligence into requirements discoveries and decisions. • Co-modeling scale shall be increased to cope with the volume and diversity of crowd. • Semi-automated and fully automated goal-modeling techniques shall be used to model CrowdRE. • Also, Business processing mining and feature modeling can be used to model CrowdRE • Annotation or Walkthrough review can be used for validating CrowdRE.
- 19. Crowd Intelligence in Requirements Engineering: Current status and future directions Prioritization and Negotiations in CrowdRE • Picking right requirements for the next release is important, which can be done through requirements prioritization and negotiation • AI argumentation, is best fit for eliminating ambiguity and decision making • Further work is required in CrowdRE for preference elicitation and win- condition elicitation. • By considering the users ranking, rating and comments about current product features. • By adopting some statistical analysis could bring the state-of-the practice to a next level of success
- 20. Crowd Intelligence in Requirements Engineering: Current status and future directions Monitoring in CrowdRE • Monitoring end users’ behavior while interacting with the software system are very essential in CrowdRE • There is existing work in monitoring end user behavior by mining user logs and mouse click events. • With the integration of crowd intelligence, we can collect feedbacks on usability issues, abnormal behaviors and run time exceptions. • Intelligent Mechanisms are required to correlate monitoring data with user feedbacks, so that developers can better interpret the user’s feedback
- 21. Crowd Intelligence in Requirements Engineering: Current status and future directions Conclusion • In this survey on crowd-based requirements engineering, • We provide an overview on the current development made on crowd-based RE and also provide a consolidated view of possible use of Crowd intelligence on crowd-based RE. • A research map is developed by mapping crowd activities to traditional RE activities in order to converge crowd intelligence in RE
- 22. Crowd Intelligence in Requirements Engineering: Current status and future directions Future Work • Our future work is to be filling the major gaps of the current research framework including active user feedback collection and analysis of user operational data for the discovery of emerging requirement. • We are using Argumentation theory to analyzed crowd requirements and identify conflict-free features underneath rationale. • Based on formal argumentation theory, negotiations arrived between crowd- users is resolved to achieve requirements decision making.