An Empirical Comparison of the Development History of CloudStack and Eucalyptus

Editor's Notes

• #3: Cloud Computing is where data, software and applications , or computer processing power are accessed from a cloud of online resources.
• #4: Rather than purchasing servers, software, data center space or network equipment, clients instead buy those resources as a fully outsourced service.
• #5: Today there are many commercial tools that provide Cloud computing services, such as amazon’s services….
• #6: Also there are many existing open-source solutions like Eucalyptus, Cloudstack, OpenStack, OpenNebula, which enable users run programs on their own resources and provides a detailed view of what would otherwise be the black box of cloud-computing services. The use of these open source has grown dramatically in recent times, So because like any other software system, cloud solutions are subject to frequent evolution, and may suffer from bugs and other quality problems that may affect the user experience. And In order to improve service quality infrastructure planning, and selection of the right software solutions. users and cloud creators need to evaluate and compare the performance of the features offered by competing platforms.
• #7: Cloud users need assurance about the availability and maintainability of the solution they are adopting, while cloud creators need to understand how their competitors are maintaining and improving their product.
• #8: That’s why we decided to focus on the quality aspects of cloud computing solutions. In this study we focused on the historical analysis and comparison of the development of this two systems. And based on Lehman’s laws of software evolution we identified our research questions.
• #9: First: Second: And lastly
• #10: In the beginning and To determine the most appropriate cloud solutions for our analysis we searched for the most popular open source projects that use the same programming language for their development, and that use the same issue and bug tracker, and that are hosted on the same version control system. And we found Eucalyptus and Cloudstack to be considered as they are both hosted on Github and use the Jira… and developped essentialy with Java.
• #11: Next, To enable historical analysis of the co-evolution of the source code and unit test code in the considered projects, we created a local clone of their GitHub repository. And we extracted all the available informations about all issues reported on the Jira issue tracker and then we analysed that .
• #12: So, first, and in order to know ……….. we analysed the commit history on GitHub for both considered projects since their first commit and until March 2017. First of all, we notice that CloudStack appears to be twice as big as Eucalyptus, even if the latter is older than the first. Eucalyptus started with a limited number of files, while CloudStack started with a big number of files since the first commit, which can be explained by the fact that CloudStack was already in development before coming to GitHub in August 2010. -Eucalyptus was split into two editions, open-core and open source. But after July 2011, when Citrix purchased Cloud.com, the number of files of the CloudStack system decreased, to start increasing again after the donation of the project to the Apache Software Foundation in April 2012. -However, Java les are the main artefact aecting the growth of both projects. For both systems, the number of Java les started with 50% of all les, and ended up to 70%
• #13: To understand how active is the community of both projects, we calculated the number of added and changed les of any type, and the Java les in particular, as well as the number of developers participating in development in each month.
• #14: For both systems we found that the edited les in most times, are les related to programming (i.e., Java les). However, we noticed that some unusual but expected activities happened after some events, related to the project management. For example, when Eucalyptus returned to be fully open source in June 2012, a large number of les was added to the project, and after CloudStack became a Top-Level Project of Apache in March 2013 and released the rst stable version after graduation, a high number of les, mostly Java les, was edited.
• #15: we focused on unit tests only, and compared test coverage in CloudStack and Eucalyptus. Both systems make use of software libraries that support unit testing. While CloudStack uses the frameworks Spring, TestNG and JUnit , Eucalyptus only uses JUnit . -To calculate the number of tested Java classes, for each Java file, excluding Java files that contain test cases, we extracted the Java classes inside and we analysed them. We observe that the number of tested Java classes in CloudStack is increasing over time. Until 2013, only 10% of Java classes were tested, but after the release of the first stable version, this number changed to achieve 74% in 2017. However, in 2013, only 1.7% of all Java files in CloudStack were test files, this number slightly changed to become 10% in 2017. -For Eucalyptus, only a small portion of Java classes were tested across the system lifetime, the maximum was in December 2016 with 4% of all classes. - These results reect that CloudStack is following a good test-driven development approach.
• #16: - In order to know how many developers participated in unit testing development, for each year we calculated for each Java test file, the number of developers that participated in its development. - We observe that the number of CloudStack developers participating in test code is high, the maximum number was in 2013 with 65 different developers. While for Eucalyptus, only a maximum of 8 different developers worked on Java test files in 2014.
• #17: We analysed the publicly available issue reports of CloudStack and Eucalyptus found on the Jira issue tracker since 2012.
• #18: - In Eucalyptus, 72% of all issues have a Major priority. 49% of these issues and 58% of all issues are bugs. With 55%, the percentage of the Major issues in CloudStack is less than in its competitor, but the percentage of bugs of this proportion is higher than in Eucalyptus with 67%, and 76% of all issues. For the other priorities, for both projects, the percentage of bugs is higher than 70%
• #19: After that we Compared between the considered projects, the distribution of the time between the creation date and the last update of a closed issue across all found issues. -And we found that for the issues related to new features and improvements, there is no signicant difference between the time that CloudStack and Eucalyptus issues take before being closed. - We also found that The bug issues in CloudStack take less time before closing them than the bugs in Eucalyptus. To conrm our observations, and to confirm that we carried out a one-sided Mann-Whitney.
• #20: We also studied the evolution of the number of bugs created over time for both Eucalyptus and CloudStack. We observed that for both projects, the number of bugs reported over time tends to decrease. The highest number of bugs was found in 2013 for CloudStack and in early 2014 for Eucalyptus. The number of bugs decreased faster in CloudStack, from 600 created bugs per month in 2013 to 50 bugs per month in 2016. This can perhaps be explained by the testing approach that CloudStack has pursued after 2013.
• #21: And next For each bug report, we extracted and calculated the number of participants. - We found that the number of participants in bug handling for Eucalyptus is higher than in CloudStack. And We statistically tested and confirmed this hypothesis with a one-sided Mann-Whitney U test
• #22: After that, and In order to analyze the bug life cycle for both considered projects in this study, for each bug we extracted the history of all state transitions of the bug report. Then we generated a generic bug life cycle with the mean time that a bug takes to remain in each state, considering the most repeated life cycles of more than 9,000 bugs in Eucalyptus and 6,000 bugs in CloudStack.
• #23: - We found that both cloud systems have dierent bug life cycles even if they use the same issue tracker - As shown in this figure, CloudStack has a generic bug life cycle with a few number of states. Most bugs in CloudStack follow the transition sequence: Open ! Resolved ! Closed.
• #24: While a typical bug life cycle in Eucalyptus adopts the following sequence: Unconfirmed ! Confirmed ! In Progress ! [In Re-view] ! In QA ! Release Pending ! Resolved ! Closed. This longer life cycle might explain why it takes longer, on average, for bug reports in Eucalyptus to reach the Closed status. It also implies that the number of participants involved in a bug x tends to be higher that for CloudStack.
• #25: Our results may not be generalisable to non-Java cloud solutions or to closed-source industrial cloud solutions that are typically subject to more restricted development rules. - In our approach we assume that a Java class is being tested if at least one of its methods is tested, which may lead to false positives, since a Java class can have many methods that together provide the functionality offered by this class.
• #26: To conclude, We analysed two popular and competitive open source cloud computing solutions developed in Java, namely CloudStack and Eucalyptus, based on data about their development history extracted from the GitHub code versioning tool and its associated Jira issue tracker.And We observed: - that, initially, CloudStack was growing differently, but in the last two years it started growing at the same rate as Eucalyptus. We also found that CloudStack has a good testing approach with a better unit testing coverage than Eucalyptus. -We also found that the studied systems have dierent bug life cycles and take dierent times to resolve and close a bug. These ndings are potentially useful to the developer communities responsible for these cloud solutions for improving development and testing activities, as well as for researchers and new contributors so they can learn from past experiences.
• #27: In future work, we will extend our comparison to include more systems like OpenStack and OpenNebula, that use different programming languages like Python and C++, and we will also take into consideration the performance and feature aspects. We will also carry out a more fine-grained bug analysis considering the messages.