Performance Comparison of Binary Machine Learning Classifiers in Identifying Code Comment Types: An Exploratory Study
0 likes54 views
This document presents an exploratory study comparing the performance of 19 binary machine learning classifiers on classifying code comments from three programming languages. The study aims to evaluate these classifiers, including logistic regression and random forest, in order to optimize and improve code comment categorization, with findings showing that all classifiers outperformed baseline scores. Key results indicate that logistic regression achieved the highest f1-score, while other models like linear SVC and decision tree also showed significant performance improvements.
Performance Comparison of Binary Machine Learning Classifiers in Identifying Code Comment Types: An Exploratory Study
- 1. Performance Comparison of Binary Machine Learning Classifiers in Identifying Code Comment Types: An Exploratory Study Amila Indika, Peter Y. Washington, Anthony Peruma
- 2. Overview As part of the NLBSE tool competition, we compare the performance of 19 binary machine learning classifiers for code comment categories that belong to three different programming languages.
- 3. Introduction • There has been a significant interest in leveraging Artificial Intelligence (AI) and Machine Learning (ML) techniques to enhance various aspects of software engineering • Traditional software engineering activities often rely on manual and rule-based approaches • which can be time-consuming, error-prone, and limited in handling complex software systems • Prior work has shown the effectiveness in using AI/ML techniques to automate many software engineering activities • Code generation and refactoring recommendations, defect detection, code comprehension and documentation, etc.
- 4. Code Comment Classification • Code comments are written in natural language and can provide information that may not be immediately obvious from the source code alone. • Grouping code comments into related categories automatically can help developers locate pertinent information faster • Prior work by Rani et al. and the NLBSE competition baseline utilized Random Forest to achieve an optimal classifier • However, there are many other types of classifiers, each with their own set of hyperparameters • Can we further optimize the Random Forest Classifier? • Are there other ML models that are better than Random Forest?
- 5. Goal, Impact & Contribution A comparison of the performance of different types of binary machine learning classifiers to understand the extent to which they can identify code comment types Help the research community understand the strengths and shortcomings of using such models for this particular task and discover avenues for future research in this area A publicly available set of binary machine learning models for code comment classification
- 7. Experiment Design • Source Dataset - a dataset of 6,738 comment sentences from 20 open-source projects implemented using Java, Python, and Pharo produced by Rani et al. • Text Preprocessing - transform the comment sentences to a standard and convenient format (i.e. ,text normalization) • Removal whitespaces; Expansion of contractions; Removal of non-alpha characters, single-character words, stopwords, convert to lowercase, stemming, replace digits and empty strings with a token
- 8. Experiment Design • Filter Category – for each programming language dataset, we filter the comments for each category • Java & Pharo – 7 categories • Python – 5 categories • Train/Test Split - The source dataset contains a column indicating whether a comment sentence belongs to the training or test set, which is used to create the train/test datasets
- 9. Experiment Design • Feature Extraction – we only utilize the comment sentence text as input to the classification model; the Term Frequency-Inverse Document Frequency approach is used to convert raw comment text into numerical values • Oversampling – we utilized random oversampling to generate new samples for under-represented classes in the training dataset; the test dataset was not oversampled
- 10. Experiment Design • Model Training & Tuning – we evaluate 8 common machine learning classification algorithms; 10-fold cross-validation to search over hyperparameter values • Naive Bayes (NB): Multinomial NB and Bernoulli NB • Support Vector Machines: Linear Support Vector Classifier • Trees: Decision Tree and Random Forest • Nearest Neighbors: K-Nearest Neighbors • Linear Model: Logistic Regression • Neural Network: Multi-Layer Perceptron • Optimized Model – the model that performs the best when built on the training data using hypermeter tuning
- 11. Experiment Design • Test Prediction – The best model for each algorithm is used to predict values for test data that the model has not seen before • Model Performance Scoring – The precision, recall, and F1 scores are calculated for each model and category. A total of 190 instances of precision, recall, and F1 scores are calculated for the evaluation.
- 12. Results
- 14. Competition Ranking Scores Model Avg. F1 % Outperformed Categories Ranking Score LogisticRegression 0.5465 1 0.6599 LinearSVC 0.5474 0.9474 0.6474 RandomForestClassifier 0.5366 0.8947 0.6261 DecisionTreeClassifier 0.4931 0.9474 0.6067 MultinomialNB 0.5249 0.8421 0.6042 MLPClassifier 0.5227 0.8421 0.6026 BernoulliNB 0.5225 0.8421 0.6024 KNeighborsClassifier 0.5033 0.8421 0.588 • All 19 Logistic Regression models outperformed the baseline scores • Our Random Forest Classifier outperforms the baseline RF model • This can be due to more exhaustive hyperparameter tuning • Alternate preprocessing activities
- 15. Summary • We examined the effectiveness of eight machine learning models to classify code comments • Workflow steps include: text preprocessing, oversampling, and hyperparameter tuning using grid search with 10-fold cross validation • All models achieve a higher average F1-Score than the baseline • Logistic Regression was the only classifier that successfully outperformed all baseline classifiers • Linear SVC and Decision Tree outperformed 18 of 19 baseline classifiers