Tizen apps with Context Awareness and Machine Learning
Download as PPTX, PDF0 likes820 views
The document discusses context-aware applications utilizing machine learning and various sensors to enhance user experiences. It elaborates on the lifecycle of contextual apps, including data sensing, understanding, and adaptation, along with the importance of user privacy and battery efficiency. The Tizen platform is highlighted for providing built-in context APIs that facilitate the development of these smart applications.
Tizen apps with Context Awareness and Machine Learning
- 1. Tizen apps with Context Awareness & Machine Learning Shashwat Pradhan 沙沃特 普瑞韩 Emberify 埃姆拜瑞菲
- 2. 2 Introduction • Context refers to information that characterizes a situation, between: – Apps – People – Surrounding environment • Contextual apps are also known as Context aware apps which understand what is going on with and around the user • Talk to other apps such as social media, email, messages
- 3. 3 Introduction • Context is about knowing the user • Current location, time, surrounding brightness, user activity, etc • Context today is being used to simplify the users life, simpler interactions & automatic sensing • Compare sensors to human senses to understand the world around • Some contextual experiences: • Alarm based on weather & traffic to work by location sensing • Phone goes on silent based on proximity to office or a movie theatre • Reminders based on travel tickets on email
- 4. 4 Introduction • Lots of new sensors in the user’s smartphone • Sensors like the accelerometer can give user activity like running or driving • Combinations of sensors can understand the user better than ever • More sensors with wearables & other IoTs
- 5. 5 Contextual Lifecycle • Sense, understand and adapt • Get data from sensors or user social networks • Build algorithms to understand the data from sensors • Adapt features & customise UX Sense Understand Adapt
- 6. 6 Contextual Lifecycle Sense Understand Adapt e.g. Smart ringtone changer Turns the phone silent in movie theatres • Senses the location of the device • Understands the place by geocoding APIs • Adapts the phone sound profile to silent
- 7. 7 Contextual lifecycle Five technology forces: • Mobile (extended to Wearables) • Social Media • Big data • Sensors (extended to IoTs) • Location-based services
- 8. 8 Context with Tizen • Tizen has a large set of in-built context APIs so the apps don’t have to do all the processing on the low level sensor data • With Tizen 2.3 Activity & Gesture recognition was introduced • Recognize & react user activities like walking, running, and in-vehicle • Recognize & react to gestures like tap, shake, snap, and tilt
- 9. 9 Sensors • Average mobile device has 7 sensors • 3 out of 5 human senses have been covered – Camera – Microphone – Capacitive screens • Sensors can help the app understand the user environment • Increase the interactive nature of the app
- 10. 10 Sensors • Tizen provides direct access to sensor data through sensor manager class • The sensor manager class can be polled at intervals by your app • Poll sensors only as often as required since they consume battery life
- 12. 12 Sensors • Construct SensorManager Class • Create a listener • Add or remove listeners with interval values • Poll sensors at intervals • Receive sensor data from event handlers at polling intervals SensorManager:: AddSensorListener() ISensorEventListener::OnDataRecieved()
- 13. 13 Sensors • Alternative to using sensor manager class is to use: • Activity recognition • Gesture recognition • Processed contextual data which will be of better quality
- 14. 14 Sensors Apart from physical sensors, the mobile has lots of user data • Contact Device API • Messaging Device API • CallHistory API • Context FW • For example movie tickets, flight tickets or entire vacation itineraries can be parsed through Emails and SMSs’ • Adding a personalized touch of context to a your application
- 15. 15 Sense • Extract the power of Social Media and Big Data through social APIs • Foursquare Places Explorer • Baidu geocoding and reverse geocoding • Sina Weibo REST API Sense the user’s digital life with social APIs
- 16. 16 Applying Machine Learning • ML algorithms learn from and make predictions on data • ML algorithms work on models have to be made based on sample inputs • Enables context prediction – which sensor data is more important
- 17. 17 Applying Machine Learning • Using a combination of sensors, Machine Learning models can be used to determine user activity • Extract sensor data and train ML models • Multiple context data used together can give more specific information about user • E.g. Accelerometer & Barometer can be used together to detect walking vs cycling Sensor Data Machine Learning Server / On-device model
- 18. 18 Applying Machine Learning • ML algorithms make sense of noisy/conflicting data from sensors • Large datasets are useful to train & fine tune Machine Learning models • ML algorithms use raw sensor data to churn out signals like high level activities
- 19. 19 Case Study • Launchify- Contextual app shortcuts app by Emberify • Context triggers • Time • Location • The app tracks when and where the user uses which apps • According to that makes predictions of which app the user needs right now • Recommends top six apps as a widget
- 20. 20 Case Study • Contextual app shortcuts by Emberify • To sense it uses geofences for home & work in addition to time • This data is stored in a SQLLite database • Depending on the current context it studies previous trends of apps based on the place and time • Adapts the algorithm based on which point of context is more r elevant for the user • Based on this it predicts which top 6 apps the user might use Learning from data and making predictions on data
- 21. 21 Experiences • What I have learnt from while building context aware systems: • Some common sense assumptions are needed in addition to the sensor data based on general human behavior to get more accuracy • Sometimes sensors can give us conflicting data • Use multiple sensors to confirm it • Common sense logic can be applied to the algorithm like repeating of a certain event occurrence before counting it since it can even be a random event
- 22. 22 Use Cases • Simplifying UX • Action based on activity or event • Lifelogging • Automatic Tracking • Quantified Self • Personal Analytics • Smart Recommendations • Personalized discovery
- 23. 23 Use Cases • Current apps can be re-imagined by adding context to them • Things will be more automatic and seamless for users • A more personal touch will be provided by adding the contextual fabric • New value propositions for the users offering developers a new market
- 24. 24 Design • New UI/UX with contextual experiences • App UI is getting less important and smart notifications is the new interface • Information as a widget or notification • Apps like Foursquare provide you the information when you need it • Eg. Tips when you reach a restaurant
- 25. 25 Design • Contextual Notification becoming a priority while designing for the wrist • Low screen estate • Minimal interaction • Input methods are limited • Making it perfect for contextual experiences
- 26. 26 Design • Context to customize user experience • Adaptive UI/UX • Based on environmental conditions • Examples of adaptive user experiences: • Dark/Light theme based on ambient light sensor • Media volume based on sound in environment based on microphone • UI according to orientation
- 27. 27 Wow factor • Wow factor in apps like Foursquare • Automatically knows which restaurant the user is at and provides recommendations • High utility features been triggered automatically through contextual triggers • Ideal contextual experience
- 28. 28 Privacy limitations • Some apps are going over the freaky line • Making users nervous with their personal information • For example Nokia’s Trapster allows the user’s location to be stalked precisely • System lacking privacy • Disclose information with a privacy policy • Should be allowed to disable the service • Encryption & security protocols if data is being stored or processed on a server
- 29. 29 Battery limitations • Data should be polled only when required • Low battery sensor polling should stop or be reduced • Share data between apps • Rather than going to the sensor every time it would be more efficient to get data through an app that just polled the data • E.g. Use location from cellular towers rather than GPS is accuracy isn’t that important
- 30. 30 Other Limitations • Machine learning algorithms aren’t perfect • Location has inaccuracy based on GPS sensor • Allow the user to correct or a manual method of insertion • E.g. Slow driving can be confused as cycling
- 31. 31 Tizen 2.4 • New context with Tizen 2.4b • Maps Service with geocoding, place discovery & routes • Context FW • Context-aware app-launching and notification rules, based on time, several device status and events, and communication events. • Contextual History APIs have been added for getting device usage statistics, eg. Which app the user uses the most • Geofence Manager
- 32. 32 Tizen 2.4 • Ideas • Contextual reminders app using places instead of time • Application stats for Quantified Self apps • Interactive games based on location
- 33. 33 Future • IoTs are bringing in new ways sense the user’s environment • With smart cars & smart homes we can get more information about the user • Apps that use context will be automatic and seamless with more sensor data • Headless apps – Running in the background, minimal user interaction
- 34. 34 Questions