Towards a Holistic Approach for Mobile Application Development in Intelligent Environments

Fachgebiet Verteilte Multimodale Informationsverarbeitung
Prof. Dr. Matthias Kranz Technische Universität München

Towards a Holistic Approach for
Mobile Application Development
in Intelligent Environments

Stefan Diewald, Luis Roalter, Andreas Möller, and Matthias Kranz
Technische Universität München

Beijing, December 9, 2011
10th International Conference on Mobile and Ubiquitous Multimedia
(MUM 2011)


General Trends for Mobile Computing
over 512,000 apps
“There’s an approx. 18 billion D/Ls
app for that.”
over 340,000 apps
over 10 billion D/Ls

§  “Always connected, personal and contextual” (finally Ubiquitous Computing!!!)
§  “Rapid development”: more and more apps being developed,
but less and less time for it
§  “Rapid innovation”: apps are getting more and more complex and
having more and more features

à better tools are needed to support better development in less time

ü  Source: Appcelerator and IDC: Mobile Developer Report, January 2011 (05.12.2011)
ü  Market numbers: http://appshopper.com/ and http://www.appbrain.com/stats/number-of-android-apps (08.12.2011)

09.12.2011 Prof. Dr. Matthias Kranz - MUM 2011 2


Motivation & Contribution

§  Improvement of the technical development process by extended
debugging capabilities

§  Improvement of the application development process by extended
prototyping capabilities

§  Improvement of the algorithm development process by extended
simulation capabilities



Outline & Structure
§  Introductory example: “MagicPhone” (Wu et al., 2010) as an example for a
mobile application for indoor interaction in intelligent environments
§  Introduction to ROS (http://www.ros.org - not our contribution, just our tool!)
and its key concepts – a robotics middleware, used also for intelligent
environments (c.f. Roalter et al., 2011)

§  Current state of the art sensor simulation & tools for mobile application
development – focus: sensor-data driven and event-driven applications and
context-aware applications in indoor environments

§  Technical details of our approach
§  Examples for mobile interaction supported by the proposed approach

§  Outlook

ü  Jiahui Wu, Gang Pan, Daqing Zhang, Shijian Li, and Zhaohui Wu. MagicPhone: Pointing & Interacting. In Proceedings of the12th
ACM International Conference Adjunct Papers on Ubiquitous Computing (Ubicomp '10). ACM, New York, NY, USA, 451-452.
ü  Luis Roalter, Andreas Möller, Stefan Diewald, and Matthias Kranz. Developing Intelligent Environments: A Development Tool
Chain for Creation, Testing and Kranz - MUM of Smart and Intelligent Environments. In Proceedings of the 7th International
09.12.2011 Prof. Dr. Matthias Simulation 2011 4
Conference on Intelligent Environments (IE '11). pp. 214-221, Nottingham, UK, July 2011


Modern Applications need Rich Sensor Data

§  Physical ubiquitous interaction §  Indoor localization / navigation
§  Context-based §  ...

Esp. indoor apps are heavily dependent on local and environment sensor data.

ü  Image Source: MagicPhone: Pointing & Interacting: http://www.youtube.com/watch?v=6aVnSmY81KA (06.12.2011)
ü  Jiahui Wu, Gang Pan, Daqing Zhang, Shijian Li, and Zhaohui Wu. MagicPhone: Pointing & Interacting. In Proceedings of the12th
ACM International Conference Adjunct Papers on Ubiquitous Computing (Ubicomp '10). ACM, New York, NY, USA, 451-452.



Testing & Development of Mobile Applications
§  Context-aware applications depend on
multiple sensors’ data at the same time
(e.g. accelerometer, gyroscope, location, …)
§  Current limitations on multi-sensor simulation
possibilities require that currently:
•  (much) testing is carried out on real devices
•  the tester has to repeat the physical actions
(incl. moving around, pointing, …) over and
over again in the real world
§  Often functions are tested separately
(e.g. only one sensor input at a time) which does
not allow verification of the complete solution
§  Testing in artificial lab environments can only cover
a very limited set of scenarios
 Virtual Prototyping as potential approach for
mobile application development in UbiComp
ü  Image Source: MagicPhone: Pointing & Interacting: http://www.youtube.com/watch?v=6aVnSmY81KA (06.12.2011)


ROS as Middleware for
Distributed Sensor-Actuator Systems
§  ROS = Robot Operating System, originates from the robotics domain, but it is
also used for intelligent and ambient assisted living environments
(intelligent environment as “ImmoBot”, static distributed sensor-actuator system)
http://www.ros.org/wiki/vmi-ros-pkg ß Our complete stacks avail. online
§  Provides hardware and software abstraction, device drivers, message passing,
visualization and simulation tools (2D and 3D), (…)

Message Passing system (additionally also a service-oriented comm. system)
§  Event-based message generation & handling (somewhat similar to Android!)
§  Basic message types e.g. for strings, integers or floats
§  Special message types e.g. for images, point clouds or object poses
§  Publisher / Subscriber model allows fast interconnection of components
§  All kinds of messages can be recorded and replayed.

Subscriber 1 Publisher Function Subscriber
Publisher msg
09.12.2011 Subscriber n
Prof. Dr. Matthias Kranz - MUM 2011 e.g. add noise, rectify, … 7


Realization of Realistic Simulation with ROS (IE 2011 Paper)

Gazebo 3D Simulator (as part of ROS)
§  Goal: (more) realistic 3D environments
with full texture and lighting support
§  Environments can be created by
using a modified interior design
application (‘SweetGazebo’)
§  Objects can be created with
common 3D modeling tools such as
Blender, Cinema4D or 3ds Max
§  Physics simulation: realistic behavior by defining
mass, inertia and friction values for objects
§  Virtual sensors and actuators can be integrated in ROS via plugins and
attached to any object in the simulation.
§  ROS connection allows reading out and setting objects’ states

ü  Luis Roalter, Andreas Möller, Stefan Diewald, and Matthias Kranz. Developing Intelligent Environments: A Development Tool
Chain for Creation, Testing and Kranz - MUM of Smart and Intelligent Environments. In Proceedings of the 7th International
09.12.2011 Prof. Dr. Matthias Simulation 2011 8
Conference on Intelligent Environments (IE '11). pp. 214-221, Nottingham, UK, July 2011


Current Situation w.r.t. the Development Support

Development Support for Sensor Data at the
Example of Android-based Devices (fastest growing OS):
§  Position: input of GPS data via e.g. the NMEA protocol
 goal: make it easier to input (esp. for indoor env.)
§  Camera: test image or test video inclusion via a
Client/Server solution – but:
- How long would the user actually point a device in a certain direction?
- How does this additional code influence the app during development, and
when is it removed later? (not transparent to the application)
à goal: make it part of the simulator/emulator!
§  (Acceleration/Pose/…) Sensor data simulation/generation:
artificially generated with an input device with mismatching dimensions
(2D mouse vs. 3D orientation), some support on replay/recording of data sets
 goal: make it more realistic



Current Situation w.r.t. the Development Support

ü  Left: Samsung Sensor Simulator, source: http://innovator.samsungmobile.com/image.view.do?serviceId=cmsCnts
&attachmentId=11105 (05.12.2011)
ü  Right: Live Camera Previews in Android, source: http://www.tomgibara.com/android/camera-source (05.12.2011)



Current Situation – OpenIntents Sensor Simulator

ü  Image Source: OpenIntents Sensor Simulator: http://code.google.com/p/openintents/wiki/SensorSimulator (05.12.2011)



Issues and Problems with Current Development Support

§  Physicality of the human as user and of the device in space is often neglected,
e.g. movement in an environment, pointing of a device at an object,
scanning/touching of a QR or NFC marker, co-location of objects, relation of
objects to each other, interrupts during application runtime as standard, not
exception, …
§  Imperfection and variability of the real world are not really taken into account:
à temporal and spatial instability/variability (of sensor data) “ignored”
(“spice” of real world challenges – c.f. impressive UbiChallenge presentations!)

Examples of what is often neglected/not investigated during mobile app development:
•  WLAN connection is ‘always bad’ at a given location (that is, the connection
breaks, like in the real world, bursty location updates/jumpy positioning/
suffering localization accuracy indoors; c.f. Kranz et al, 2010),
•  places change frequently (like posters at the walls that influence tracking), …
ü  M. Kranz, C. people/objectsComparativethe line of sight of a camera, …Localization. 8th IEEE International
•  Fisher, A. Schmidt. A walk in Study of DECT and WLAN Signals for Indoor
Conference on Pervasive Computing and Communications (PerCom2010). pp. 235-243, Mannheim, Germany, March 2010



Room for Improvement in the Development Process
§  Debugging & testing of interactive systems already during development phase
(and not at the end) à not really available as of now (c.f. reviewed simulators)
§  Still no (little) realistic statements can be made on usage/performance of apps in
real(istic) environments as they are (mostly) not part of the development
process!
à not really (quickly) testing possible in multiple environments before
deployment/extensive real world testing
§  Complex interactions such as touching objects or grabbing an image of a
display can currently not directly be investigated in the development tools
à not really available as of now

Challenges:
§  How to provide (better) data (or at least initial data at all)?
§  How to more efficiently develop those complex interactive systems (and not only
mobile phone software)?
§  How to more efficiently prototype multimodal and multimedia systems faster?


Proposed Idea: Extended IDE and Development Tool Chain



Example: Virtual Camera and Virtual Display Functionality

§  Linked ROS sensors and Android sensors in the example: camera and
accelerometer.
§  Display: Android output that is pushed back to the simulated 3D environment
 all simulation no longer part of the application, but the emulator/simulator
(question: how do we move/steer/control the device in the simulation? à later!)



Supported Sensors and Actuators

Sensors
§  Accelerometer* and Gyroscope
§  Camera* and Depth Image Sensor
§  Light Sensor
§  Proximity Sensor
§  GPS
§  NFC Reader
§  Battery Status A virtual Android device in a
realistic 3d simulation of an office.
§  …
Actuators
§  Light Emitting Diodes and Displays*
§  Projectors
§  Speakers * Currently fully supported by the connected Android emulator,
all others are already supported by ROS as well
§  …



Example 1: Indoor Localization and Physical Interaction
using NFC tags and QR codes

Real Simulated
§  Connect device, install changed §  Install changed version of app
version of app, disconnect device §  Perform test: replay recorded data set
§  Perform test: walk several meters in (even faster than real-time), variation
corridor, take picture or touch NFC can be added automatically,
tag e.g. position offset, …


Ex 2: Indoor Localization/Tracking using Visual Information

Instead of taking various fixed and prerecorded videos, daylight, obstacles, and
other properties are simulated in an extended IDE/tool chain.
ü  PTAM (Parallel Tracking and Mapping), source: http://www.robots.ox.ac.uk/~gk/PTAM/ (06.12.2011)
(markerless, feature-based tracking, no a-priori maps, no inertial measurement unit; test implementation)


Example – Manual Testing of Mobile Applications in the
Virtual Environment

§  3d input devices like the 3Dconnexion SpaceNavigator
or joysticks can be used for 6 degrees-of-freedom
controlling

§  Mobile device’s accelerometer,
gyroscope and magnetometer for 3D

+
orientation + classic keyboard for 2D
navigation (walking through the virtual
scene by holding a real Android device
for the pose information and steering by
the keyboard for the 2D location info.)

ü  Image Sources: top: 3Dconnexion – SpaceNavigator: http://www.3dconnexion.de/products/spacenavigator.html (06.12.2011),
bottom left: http://alwaysinternet.de/wp-content/uploads/2010/12/google-nexus-s-samsung.jpg (06.12.2011),
bottom right: http://www.logitech.com (06.12.2011)



Example – (human-inspired robotic) Path Planner and
Navigation Stack for Automated (Randomized) Testing

§  ROS offers several path planning and
navigation stacks (spice for getting slightly
different paths on each walk from A to B)
§  Autonomic ROS running robots (like the PR2)
with laser scanner and/or depth cameras can
be used for generating a map
§  Random paths can be created by imposing
different constraints for the route
§  Mobile device could be steered along path
e.g. having a fixed pose similar to holding a
mobile phone in front user
§  Test how applications can adapt to
disturbances, variations, …

ü  Image Source: ROS package - base planner: http://www.ros.org/wiki/base_planner_cu (07.12.2011)



Example – Testing with different Light Conditions

§  Light conditions can be varied in the 3d
simulation.
§  Light direction, color and intensity can be
adjusted.
§  Shadows and reflections are supported.
§  Additional distortion can be applied to the
virtual camera image.

Can be used for testing image/vision-based
applications.

Example: Checking under which conditions or
distances a designed marker is still readable.



Summary of the Potentials of the Proposed Approach

§  Multi-sensor simulation in a (more) realistic & easy-to-create 3d
environment.
§  Sensor data can either be generated automatically, or by a human moving
the mobile device in the intuitive 3D simulated virtual environment.
§  ‘Randomized’ automatic test cases can make use of available components
(e.g. route planning algorithms) and data sets (biasing of positions and other
sensor values)
§  Recording and replaying of all sensors and states (via ROS messages)
allows detailed debugging.
§  Connection to a hardware emulator allows debugging end-user versions
of mobile applications (no additional source code necessary, that is, no added
code that changes the app’s behavior)
§  Pushing back the mobile’s displays’ content enables realistic usage of
mobile applications in the simulated environment



Outlook – To Dos for the Future

§  Verification and analysis of the tool chain / approach:
•  Survey and test with professional developers is planned
•  Compare development time within the scope of our research-oriented
teaching: students will develop various indoor localizations systems with
and without tool chain support
•  Analyze and compare the types and amount of errors made with normal
and proposed approach
§  Evaluation of realism of the simulated sensors: metrics need to be defined
§  Exploration of new interaction potentials: fast prototyping of mobile interaction
with e.g. robots and intelligent environments (home automation systems),
possible due to their ROS support
§  Extension for other (physical) interaction approaches, esp. Tangible User
Interfaces



Outlook – Further possible Application Areas

§  Public and private displays
§  2d and 3d touch interfaces
§  Tangible User Interfaces (I like cubes)



Outlook – 2D Tabletop Interactions with a Mobile Device

Mixed physical-digital interactions:
§  Displays: either external or mobile device as pocket table
§  Physical Objects (e.g. Tangible User Interfaces)
§  NFC (Near Field Communication) tagged objects

ü  Image Source: reacTable: http://www.reactable.com/ (07.12.2011)



Questions?

? ?
? ??

09.12.2011
Thank you- MUM 2011 much for your kind attention!
Prof. Dr. Matthias Kranz
very 26

Prof. Dr. Matthias Kranz www.vmi.ei.tum.de
Technische Universität München


m.de
i.ei.tu m.de
w ww.vm nz@tu
atthia s.kra
m

09.12.2011
Thank you- MUM 2011 much for your kind attention!
Prof. Dr. Matthias Kranz
very 28


Combination of the Real and Simulated World at the
Example of Indoor Localization Systems

Environment

Real Simulated
Real world test, most
Ground truth, testing of
Real effort and time
controlled effects
consuming
Mobile Device
Real effects, but no Fast evaluation of
Simulated working prototpye algorithms, everything
necessary controllable

ROS can be used to connect real and simulated components.



Thank you for your attention!
Questions?

?
?
matthias.kranz@tum.de
www.vmi.ei.tum.de/team/matthias-kranz.html

04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 30


Paper Reference

•  Please find the associated paper at:
https://vmi.lmt.ei.tum.de/publications/2011/MUM11_preprint_v2.pdf

•  Please cite this work as follows:
•  Stefan Diewald, Luis Roalter, Andreas Möller, Matthias Kranz. 2011.
Towards a Holistic Approach for Mobile Application Development in Intelligent
Environments.
In: Proceedings of the 10th International Conference on Mobile and
Ubiquitous Multimedia (MUM 2011), Beijing, China, pp. 73-80, December
2011



If you use BibTex, please use the following entry
to cite this work:

@InProceedings{MobileSimulation11,
author = {Stefan Diewald and Luis Roalter and Andreas M{"o}ller and Matthias Kranz},
title = {{Towards a Holistic Approach for Mobile Application Development in Intelligent Environments}},
booktitle = {Proceedings of the 10th International Conference on Mobile and Ubiquitous Multimedia},
series = {MUM '11},
location = {Beijing, China},
publisher = {ACM},
pages = {73--80},
numpages = {8},
month = dec,
year = {2011},
address = {New York, NY, USA},
ISBN = {978-1-4503-1096-3},
url = {http://doi.acm.org/10.1145/2107596.2107604},
doi = {10.1145/2107596.2107604},
acmid = {2107604},
keywords = {Android, ROS, intelligent environment, middleware, mobile devices,
simulation, smart phones, virtual environment},
}


Towards a Holistic Approach for Mobile Application Development in Intelligent Environments

More Related Content

Similar to Towards a Holistic Approach for Mobile Application Development in Intelligent Environments (20)

More from Distributed Multimodal Information Processing Group (11)

Recently uploaded (20)

Towards a Holistic Approach for Mobile Application Development in Intelligent Environments