Conducting Experiments on the Software Architecture of Robotic Systems (QRARSAC@ICRA 2023)

Conducting Experiments on the Software
Architecture of Robotic Systems
Ivano Malavolta
i.malavolta@vu.nl

Ivano Malavolta - Vrije Universiteit Amsterdam
Software architecture
Green software
Empirical software engineering
Introductions: who am I?
If you think good architecture is expensive,
try bad architecture.
... Brian Foote and Joseph Yoder

The S2 group @VUAmsterdam and its collaborators
I/O Magazine, Issue 3, Dec. 2020

Experiment 1
Green tactics for robotics software
Ivano Malavolta, Katerina Chinnappan, Stan Swanborn, Grace Lewis, Patricia Lago. Mining the ROS ecosystem for
Green Architectural Tactics in Robotics and an Empirical Evaluation. In Proceedings of the 18th International
Conference on Mining Software Repositories (MSR), 2021. [PDF]
Katerina Chinnappan, Ivano Malavolta, Grace Lewis, Michel Albonico, Patricia Lago. Architectural Tactics for
Energy-aware Robotics Software: A Preliminary Study. In Proceedings of the European Conference on Software
Architecture (ECSA), 2021. [PDF]

Architectural tactics
A tactic is a design decision that refines a high level style and is influential in the
control of a quality attribute response
ROS style
pub/sub + parameters server + services + ...
Design
stimulus

Example - Heartbeat

Example - Cloud-only obstacle detection

Example of tactic:
(local + cloud)-based SLAM
Motivation
https://github.com/googlecartographer/cartographer

Example of tactic:
(local + cloud)-based SLAM
Description
https://github.com/googlecartographer/cartographer

Goal
To identify and
empirically evaluate
architectural tactics for
energy-efficient robotics software
By mining the ROS ecosystem to identify the
green tactics
By conducting an experiment on a real robot
for evaluating the identified green tactics
How

Green tactics identification
https://github.com/S2-group/msr-2021-robotics-gre
en-architectural-tactics-replication-package
Replication package

Green tactics for robotics software
Limit task
(EE1)
Disable
hardware (EE2)
Energy-aware
sampling (EE3)
On-demand
components
(EE3)

Empirical evaluation of the green tactics
Name Treatments
Tactic - Baseline (no tactics applied)
- Tactic EE1
applied
- Tactic EE2
applied
- Tactic EE3
applied
- Tactic EE4
applied
- All tactics applied
Movement - No movement
- Fixed movement
- Autonomous movement
Environment - Empty
- Obstacles
Independent variables
30 trials
10 Runs per trial
300 runs = ~10 Hours
Dependent variable
energy consumption in J during the whole mission
Arduino Nano + INA 219 sensor
Mission
To move at 0.6m/s within the arena for 2 minutes while:
(1) continuously video-recording at 60 FPS and
(2) stopping and doing a 360° rotation every 20s

Empirical evaluation of the green tactics
1. On average, the application of the
green tactics improve energy
efficiency
a. However, not always with the same
magnitude
2. The combination of all tactics (C)
improves energy efficiency more than
each tactic in isolation
3. Green tactics reduce energy consumption
also across different movements and
environments

Stan Swanborn, Ivano Malavolta. Robot Runner: A Tool for Automatically Executing Experiments on Robotics
Software. In Proceedings of the ACM/IEEE 43rd International Conference on Software Engineering (ICSE), 2021. [PDF]
Robot Runner
https://github.com/S2-group/robot-runner

Robot-runner
Event-based orchestration
class RobotRunnerConfig:
# Name for this experiment
name: str = "new_robot_runner_experiment"
# Required ROS version for this experiment to be ran with
# NOTE: (e.g. ROS2 foxy or eloquent)
# NOTE: version: 2
# NOTE: distro: "foxy"
required_ros_version: int = 2
required_ros_distro: str = "foxy"
# Experiment operation types
operation_type: OperationType = OperationType.AUTO
# Run settings
time_between_runs_in_ms: int = 1000
# Path to store results at
# NOTE: Path does not need to exist; appended with 'name' and created on runtime
results_output_path: Path = Path("~/Documents/experiments")
Self-contained Python config file
__run_id __done Task Repetition CPU Memory Energy
run_0 DONE networking 5 17.256701030927836 23.87651775486827 863868.5
run_1 DONE computation 2 25.5279123414071 23.2745098039216 887002
run_2 DONE video 1 19.3591385331781 24.6903376018626 884413.3
run_3 DONE video 4 19.230303030303 24.7420745920746 871886.5
run_5 DONE computation 3 25.7290322580645 23.9 856502.8
run_7 DONE video 3 19.6579493087558 24.0344470046083 869336.1
run_9 DONE video 5 20.3076388888889 24.6690972222222 792734.8
run_11 DONE video 2 19.0021889400922 24.0443548387097 872746.5
run_13 DONE networking 4 17.03945267958951 23.86864310148233 878283
def populate_run_data(self, context: RobotRunnerContext) -> dict:
variation = context.run_variation
metrics_location = str(context.run_dir.absolute()) + '/metrics.txt'
energy_location = str(context.run_dir.absolute()) + '/energy.txt'
df_metrics = pd.read_csv(metrics_location, sep=",", header=None)
df_energy = pd.read_csv(energy_location, sep=",", header=None)
variation['CPU'] = df_metrics['CPU'].mean()
variation['Memory'] = df_metrics['Memory'].mean()
variation['Energy'] = auc(df_energy['Power'])
return variation

Run ‘em all!
https://github.com/S2-group/android-runner https://github.com/S2-group/experiment-runner
Android (web) apps Generic

Runners
Our Runners as a learning platform
Green Lab Final projects
● Green Lab = Master course on
empirical software engineering
for energy efficient software
● Students use our Runners as
black-box tools for their own
experiments
● Students go deeper on
run-time profiling
○ Plugins for new profilers
○ Improve Runners
● Community of learners
● Discover and fix bugs
● Runners always up to date
● Learn the basics of OSS development

Experiment 2
Computation offloading for ground robots
Milica Ðorđević, Michel Albonico, Grace A. Lewis, Ivano Malavolta, Patricia Lago. Computation Offloading
for Ground Robotic Systems Communicating over WiFi – An Empirical Exploration on Performance
and Energy Trade-offs. In Empirical Software Engineering journal, 2023. [PDF]

Experiment definition
V. Basili, G. Caldiera, and H. Rombach. Goal question metric paradigm.
Encyclopedia of software engineering, 1:528–532, 1994.

Meet
Sherlock

Offloadable tasks
● Simultaneous localisation and mapping (SLAM)
○ Gmapping - http://wiki.ros.org/gmapping
● Navigation and path planning
○ ROS Navigation stack - http://wiki.ros.org/navigation
● Object recognition
○ Find_object_2d - http://wiki.ros.org/find_object_2d

Hardware instrumentation
● Hardware
○ TurtleBot3 Burger
○ INA219 power profiling
circuit (200Hz)
○ Remote PC
● Software
○ Robot Runner
○ PSUtil (50Hz)
○ Wireshark
TurtleBot3 Burger INA219 power proﬁling circuit
SD card module
INA219
sensor
Arduino
NANO

Metrics
Metric Unit of measure Description
Power Consumption mW
The instantaneous power consumption sampled at 200Hz during the entire mission execution.
The energy consumption (in Joules) is then calculated as the integral of power consumption
(mW) over the mission execution time.
CPU usage % The average CPU usage during the entire mission execution.
RAM utilization Mb The average RAM utilization during the entire mission execution.
Number of network
packets
Count
The total number of network packets exchanged between the robot and remote PC during the
mission execution.
Size of network packets Mb
The total size of network packets exchanged between the robot and remote PC during the
mission execution.
Feature extraction time ms The average time needed for the features to be extracted from a received image frame.
Object detection time ms
The average time needed for comparing the extracted features against images in the database
to conclude if any of the objects in the database are recognized in a received image frame.
Detection result delay ms
The average delay of transferring object detection outcome from the object recognition node to a
node that logs the recognition result on the robot.
Navigation time ms The average time it takes for a robot to navigate from its current location to a goal location.
Mission execution time ms The total duration of the mission.

ROS computation graph

Measurement setup
Robotic
system
Remote PC
Robot Runner
Run table
INA219 profiler client
CPU/Memory
profiler client
INA219 profiler
server
CPU/Memory
profiler server
Mission
SLAM configuration
Experiment
configuration
Power profiling
circuit
Run folders
SLAM algorithm
Turtlebot3
bringup
SSH Client
SSH Client
Rosbags and logs

Results (excerpts)
Energy consumption

Results (excerpts)
Mission execution time

Results (excerpts)
CPU and memory usage

Results (excerpts)
Network traffic

Offloading - Takeaway messages

Engel Hamer, Michel Albonico, Ivano Malavolta. Resource Utilisation of 2D SLAM Algorithms in
ROS-Based Systems: an Empirical Evaluation. Robotics and Autonomous Systems journal (under review),
2023.
Experiment 3
Resource utilization of 2D SLAM algorithms

Goal
Conceptual
level
Analyse SLAM algorithms for the purpose of evaluating how they impact resource utilisation from the point of view of robotics researchers and practitioners in the
context of ROS-based systems.
Metrics
Quantitative
level
Questions
Operational
level
RQ1.1: How do SLAM
algorithms impact
energy consumption
of ROS-based
systems?
RQ1.2: How do SLAM
algorithms impact CPU
utilisation of ROS-based
systems?
RQ1.3: How do SLAM
algorithms impact
memory utilisation of
ROS-based systems?
RQ1: How do SLAM algorithms impact resource utilisation of ROS-based systems?
Power Execution time
CPU utilisation Memory utilisation
ROS msg
count
ROS msg
size
Occupancy Corner count
Enclosed area
count
RQ2: What are the trade-offs between resource
consumption and the accuracy of generated maps
for SLAM algorithms in ROS-based systems?
RQ1.4: How do SLAM
algorithms impact
message traﬃc in
ROS-based systems?
Energy consumption
Map
Wohlin, Claes, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. Experimentation in
software engineering. Springer Science & Business Media, 2012.

Example of mission
● Two arenas
○ Internal objects
○ Loop closure
● Mission sequence
○ Start rosbag recording
○ Initialise SLAM
○ Wait 5 seconds
○ Traverse and map arena
● No communication with PC
○ All terminal output logged to disk
○ Monitoring experiment
Arena design
Real-world arena

Results (excerpt)
Memory usage
Point to point Circular
All algorithms increase over time
Hector largest relative increase (+6.5%)
Point to point
● Cartographer (+5.5%)
● Hector (+19.5%)
● Gmapping at a low map resolution and Karto are
not significantly different
● Gmapping at a low and high map resolution are
not significantly different (+1.2%)
Circular
● Cartographer (+5.56%)
● Hector (+19.9%)
● Gmapping at a low map resolution and Karto are
not significantly different
● Gmapping at a low and high map resolution are
not significantly different (+0.8%)

Overall - relative ranking of SLAM algorithms
(lower is better)
Energy
consumption
CPU usage
Memory
usage
ROS
messages
Map quality
Cartographer 2 1 3 3 4
Gmapping 1/2* 1/2* 2 2 2
Hector 1 1 4 1 3
Karto 1 1 1 1 1
*depends on the speciﬁc conﬁguration

Conclusions

Conducting Experiments on the Software Architecture of Robotic Systems (QRARSAC@ICRA 2023)

More Related Content

Similar to Conducting Experiments on the Software Architecture of Robotic Systems (QRARSAC@ICRA 2023) (20)

More from Ivano Malavolta (20)

Recently uploaded (20)

Conducting Experiments on the Software Architecture of Robotic Systems (QRARSAC@ICRA 2023)