COMP 4026 Lecture 5 OpenFrameworks and Soli
3 likes925 views
This document outlines a lecture on advanced human-computer interaction technologies, focusing on wearable computing, augmented reality, virtual reality, and the Project Soli radar-based motion sensing technology. It details the process for a class project involving advanced technology design and covers OpenFrameworks as a toolkit for creative coding in C++. Key concepts discussed include the capabilities of wearable computers, the definition of augmented reality, and the programming nuances of C++ related to OpenFrameworks.
![Augmented Reality Definition
• Defining Characteristics [Azuma 97]
• Combines Real andVirtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/85/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-11-320.jpg)
![Early Examples
• Interaction without devices:
• BodySpace [Strachan 2007]: Functions to body position
• Abracadabra [Harrison 2007]: Magnets on finger tips
• GesturePad [Rekimoto 2001]: Capacitive sensing in clothing
• Palm-based Interaction
• Haptic Hand [Kohli 2005]: Using non-dominant hand in VR
• Sixth Sense [Mistry 2009]: Projection on hand
• Brainy Hand [Tamaki 2009]: Head worn projector/camera](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/85/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-14-320.jpg)
![ImaginaryPhone
• Gustafson, S., Holz, C., & Baudisch, P. [2011]](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/85/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-15-320.jpg)
![Invisible Interfaces – Gestures in Space
• Gustafson, S., Bierwirth, D., & Baudisch, P. [2010]
• Using a non-dominant hand stabilized interface.](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/85/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-17-320.jpg)
![Class Extending
• Take one class and add functionality to it with a new class
• Eg enemy class for video game
!class Enemy {!
! !int x, y; //position!
! !.. .. !
! !public void draw() {!
! !//draw my picture to the screen at the proper location }!
!}!
• Want to draw enemy twice – create new class
!//on a "DoubleEnemy.h" file!
!class DoubleEnemy: public Enemy // class[className]:[privacy][extended Class]{}!
!{!
! !public void draw();//the actual code inthe "DoubleEnemy.cpp" file!
!}; // note the ";" at the end of the class statement!](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/85/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-48-320.jpg)
COMP 4026 Lecture 5 OpenFrameworks and Soli
- 1. LECTURE 5: OPENFRAMEWORKS AND SOLI COMP 4026 – Advanced HCI Semester 5 - 2016 Mark Billinghurst University of South Australia August 25th 2016
- 2. RECAP
- 3. Advanced Interface Technology • Wearable Computers • Augmented Reality • Virtual Reality • Invisible Interfaces • Environment Sensing • Physiological Sensing
- 4. Class Project 1. Pick Advanced Technology 2. Brainstorm use case 3. Develop conceptual design 4. Prototype interface/experience design 5. Conduct user evaluation 6. Repeat steps 3-5 7. Write report
- 5. Wearable Computing ▪ Computer on the body that is: ▪ Always on ▪ Always accessible ▪ Always connected ▪ Other attributes ▪ Augmenting user actions ▪ Aware of user and surroundings
- 7. Google Glass
- 10. 1977 – StarWars
- 11. Augmented Reality Definition • Defining Characteristics [Azuma 97] • Combines Real andVirtual Images • Both can be seen at the same time • Interactive in real-time • The virtual content can be interacted with • Registered in 3D • Virtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
- 12. Virtual Reality • ImmersiveVR • Head mounted display, gloves • Separation from the real world
- 13. AR vsVR
- 14. Early Examples • Interaction without devices: • BodySpace [Strachan 2007]: Functions to body position • Abracadabra [Harrison 2007]: Magnets on finger tips • GesturePad [Rekimoto 2001]: Capacitive sensing in clothing • Palm-based Interaction • Haptic Hand [Kohli 2005]: Using non-dominant hand in VR • Sixth Sense [Mistry 2009]: Projection on hand • Brainy Hand [Tamaki 2009]: Head worn projector/camera
- 15. ImaginaryPhone • Gustafson, S., Holz, C., & Baudisch, P. [2011]
- 17. Invisible Interfaces – Gestures in Space • Gustafson, S., Bierwirth, D., & Baudisch, P. [2010] • Using a non-dominant hand stabilized interface.
- 18. Project Soli • Using Radar to support free-hand spatial input
- 19. Google Tango • Tablet based system • Android OS • Multiple sensors • RGBD Sensor • IR Structured light • Inertial sensors • High end graphics • Nvidia tegra chip
- 20. Physiological Sensors • Sensing user state • Body worn devices • Multiple possible sensors • Physical activity • Eye tracking, gaze • Heart rate • GSR • Breathing • Etc
- 21. Tobii Eye Tracker • Wearable eye tracking system • Natural data capture • Scene camera capture • Recording/streaming eye gaze, 60 Hz sampling
- 22. OPENFRAMEWORKS
- 23. OpenFrameworks (www.openframeworks.cc) • Open source toolkit designed for creative coding • Developed by Z. Lieberman,T.Watson and A. Castro • Framework – collection of libraries • Written in C++ • More powerful than Processing, but more complicated • Must use IDE for development • Xcode,Visual Studio, Code::Blocks • Runs on Mac,Windows, Linux platforms
- 26. Why use oF instead of Processing • Speed • Accessibility of low level information • Debugger • C++ • Version control • Cross Platform
- 27. OpenFrameworks vs.Processing • Making project visible on Internet - Processing • Make a project with lots of 3D graphics - OpenFrameworks • Make a project for lots of different computers/OS – Processing • Make a project using an external library like the OpenCV computer vision library – OpenFrameworks • Make a project that interfaces with the Arduino board - Either
- 28. OpenFrameworks Installation • addons: added libraries from user community. Must be explicitly included in programs using them • apps: store your programs here.Also contains example code. • libs: where the core libraries of OpenFrameworks are stored. Also contains core openFrameworks folder
- 40. testApp.cpp
- 44. Typical FunctionTypes • setup( ) • load assets • Initialize values • Initialize addons or components • update( ) • calculations • increment video frames • draw( ) • draw shapes/images/videos • use GLSL Shaders
- 47. Classes in C++ • C++ classes comprise of two files. It helps to think of these two files as a recipe. • The header file (.h) is like the list of ingredients, and contains: • Any preprocessor statements there to prevent multiple header definitions • Any include statements to other classes • Any class extension statements • Any variables local to the class • Prototypes of any functions to be contained in the class • Security settings of these functions and variables (e.g. public, private, protected, etc). • and a body file (.cpp) which is like the instructions on what to do with the ingredients and contains: • An include statement that references the .h file • All of the code to fill in the function prototypes.
- 48. Class Extending • Take one class and add functionality to it with a new class • Eg enemy class for video game !class Enemy {! ! !int x, y; //position! ! !.. .. ! ! !public void draw() {! ! !//draw my picture to the screen at the proper location }! !}! • Want to draw enemy twice – create new class !//on a "DoubleEnemy.h" file! !class DoubleEnemy: public Enemy // class[className]:[privacy][extended Class]{}! !{! ! !public void draw();//the actual code inthe "DoubleEnemy.cpp" file! !}; // note the ";" at the end of the class statement!
- 49. Pass byValue vs.by Reference • void functn(int num) – pass by value • void functn(<class> test) – pass by reference • sends address of where class stored • use pointers to pass arrays back and forth through functions int num = 5; value stores address of variable value
- 50. void setup() { int num = 1; addOne(num); print(num); } void addOne(int num) { num++; } class Test { int num=0; } void setup() { Test test = new Test(); test.num=1; addOne(test); print(test.num); } void addOne(Test test) { test.num++; } Pass by Value Pass by Reference
- 51. & and * • In C++ you need to explicitly state whether you are passing something by value or by reference. • Use & (referencing) and * (dereferencing) symbols • the & symbol is used to acquire the memory address of a variable or function b=1;! a = &b; // a now equal to memory address of b! a++; // memory address of b + 1! *a++; // value a +1 (increments b as well)!
- 52. Example • What does this code do? ! ! !int x;! ! ! !int *ptr;! ! ! ! !x=5;! ! ! !ptr = &x;! ! ! !*ptr = 10;!
- 53. 2D Image Functions • Colors • ofFill(); • ofCircle(100,400,80); • ofSetHexColor(0x000000); ofSetColor(255,0,0,127); • Primitives • ofCircle(100,400,80); • ofRect(400,350,100,100); • ofLine(600,300,800, 250); • ofDrawBitmapString("rectangles", 275,500);
- 55. Circle Grid • Setting the size of the window. • Processing: • size(800, 600, OPENGL); • openFrameworks: • ofSetupOpenGL(&window, 800, 600, OF_WINDOW); • function is called in main() in the file main.cpp.
- 56. Circle Grid • DeclaringVariables • Processing: • Declare the variables you need right after you import the libraries you need. • openFrameworks: • Declare variables in the file testApp.h, after the line void windowResized(int w, int h);.
- 57. Circle Grid • Background Color • Processing: • background(0); will set the background of your sketch to black. You need to call the function inside draw() to draw the background each frame. • openFrameworks: • Call ofBackground(0, 0, 0); once inside the setup() method. openFrameworks will draw the background automatically each frame.You can disable this by calling ofSetBackgroundAuto(false) within setup() in the file testApp.cpp.
- 58. Circle Grid • Drawing Circles • Processing: • after you have set the stroke and fill, use ellipse(50, 50, 20, 20); to draw a circle with a diameter of 20 at (50, 50). • openFrameworks: • you can use ofCircle(50, 50, 10); to draw the same circle.You could also use ofEllipse(50, 50, 20, 20);. If you want to draw a circle with a stroke you will need to call the function to draw the circle two times. Once for the fill and once for the stroke. ofSetColor(255, 255, 255);! ! !ofFill();! ! !ofCircle(50, 50, 20);!
- 59. Graphics Demo - graphicsExample • setup( ) method • draw( ) method
- 60. Drawing Polygons • Must begin and end a shape • ofVertex, ofCurveVertex, ofBezierVertex !ofBeginShape();! ! ! !ofVertex(200,135);! ! ! !ofVertex(15,135);! ! ! !ofVertex(165,25);! ! ! !ofVertex(105,200);! ! ! !ofVertex(50,25);! !ofEndShape();! •
- 61. polygonExample
- 62. ofBoxDemo
- 63. Importing Libraries • Large set of oF addon libraries (> 450) • http://ofxaddons.com/ • Just download library to addons directory, then include library #include “myLibrary.h”! • Sample libraries • ofxOpenCv • ofxVectorGraphics • ofxVectorMath • ofxNetwork • ofvOsc
- 64. Examples • VectorGraphicsExample • 3DModelLoaderExample • Loading 3D models • assimpleExample • 3D animation • openCVExample • Hand segmentation
- 65. Projects
- 68. Nike + Paint With Your Feet +openFrameworks +GPS
- 69. Resources • Main website • http://www.openframeworks.cc/ • Forum • http://forum.openframeworks.cc/ • Addons • http://ofxaddons.com/ • Roxlu s website • http://www.roxlu.com/
- 70. PROJECT SOLI
- 71. Overview • Soli uses radar to detect fine scale finger motion
- 74. Radar Fundamentals • Radar tracks moving objects • Measures response to Radar waves sent from transmitter
- 75. Radar Reflections from Hand • Multiple reflection points
- 76. Signal Processing • Signal received combination of slow and fast time
- 78. Processing Pipeline • From raw hand motion to recognized gestures
- 79. Soli Hardware • Miniaturized Radar
- 80. Signs vs. Actions • Soli recognizes hand actions
- 81. Virtual Tools • Use virtual tool metaphor • Change with distance
- 82. Types of Virtual Tools • asdasf
- 83. Basic Gesture Movement • Easily recognize distinctive gesture motions
- 84. • > 90% accuracy on filtered results (Bayesian Filter) Recognition Results
- 85. Applications • Gesture interaction with objects • Smart watch, car console • Gesture interaction with environment • Furniture, walls • Other applications • Material recognition, Gaming, Object scanning
- 89. Future Research • Radar Sensing • Radar clutter, multi-path reflections, occlusion, etc • Machine Learning • New gesture recognition approaches • Human Factors • Measuring human performance abilities, requirements • Interaction Design • New interaction modalities, metaphors
- 90. Background Reading Lien, Jaime, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. "Soli: ubiquitous gesture sensing with millimeter wave radar." ACM Transactions on Graphics (TOG) 35, no. 4 (2016): 142.