426 lecture 4: AR Developer Tools

COSC 426: Augmented Reality

Mark Billinghurst
mark.billinghurst@hitlabnz.org

August 1st 2012

Lecture 4: AR Developer Tools

Low Level AR Libraries
  ARToolKit
  Marker based tracking
  FLARToolKit
  Flash version of ARToolKit
  SSTT
  Simple Spatial Template Tracking
  Opira
  Robust Natural Feature Tracking

What is ARToolKit?
  Marker Tracking Library for AR applications
  Open Source, Multi-platform (Linux, Windows, MacOS)
  Overlays 3D virtual objects on real markers
  Uses single tracking marker
  Determines camera pose information (6 DOF)
  ARToolKit Websites
http://www.hitl.washington.edu/artoolkit/
http://artoolkit.sourceforge.net/

ARToolKit Software
  ARToolKit version: 2.65 or later
  Currently two license models
  Open Source (GPL): ARToolKit 2.72
  Commercial (ARToolWorks): ARToolKit 4.0
  OS: Linux, Windows, MacOS X, iPhone/Android
  Programming language: C
  Related software
  ARToolKit Professional: Commercial version
  ARToolKitPlus: Advanced version
  NyARToolkit: Java and C# version
  FLARToolKit: Flash version

ARToolKit Family

ARToolKit NFT
ARToolKit
ARToolKit Plus

JARToolKit (Java)

ARToolKit (Symbian)
FLARToolKit (Flash)
NyToolKit
- Java, C#,
- Android, WM
FLARManager (Flash)

ARToolKit Structure

ARvideo.lib

DirectShow

  Three key libraries:
  AR32.lib – ARToolKit image processing functions
  ARgsub32.lib – ARToolKit graphics functions
  ARvideo.lib – DirectShow video capture class

Additional Software
  ARToolKit just provides tracking
  For an AR application you’ll need more software
  High level rendering library
  Open VRML, Open Inventor, osgART, etc
  Audio Library
  Fmod, etc
  Peripheral support

What does ARToolKit Calculate?

  Position of makers in the camera coordinates
  Pose of markers in the camera coordinates
  Output format
  3x4 matrix format to represent the
transformation matrix from the marker
coordinates to the camera coordinates

From Marker To Camera
  Rotation & Translation

TCM : 4x4 transformation matrix
from marker coord. to camera coord.

An ARToolKit Application
  Initialization
  Load camera and pattern parameters
  Main Loop
  Step1. Image capture and display
  Step2. Marker detection
  Step3. Marker identification
  Step4. Getting pose information
  Step5. Object Interactions/Simulation
  Step6. Display virtual objects
  End Application
  Camera shut down

Sample ARToolKit Applications
 Ex. 1: Simple video display
 Ex. 2: Detecting a marker
 Ex. 3: Using pattern
 Ex. 4: Getting a 3D information
 Ex. 5: Virtual object overlay

Ex 1: Simple Video Display
  Program : sample1.c
  Key points
  Loop structure
  Video image handling
  Camera parameter handling
  Window setup
  Mouse and keyboard handling

Sample1.c – video initialization
  Configure the video input
vconf = <video configuration string>
  Start video capture
arVideoCapStart();
  In init(), open the video
arVideoOpen( vconf );
arVideoInqSize(&xsize, &ysize);
  When finished, close the video path
arVideoCapStop();
arVideoClose();

Changing Image Size
  For input capture "
vconf = “videoWidth=320,videoHeight=240";
Note – the camera must support this image size

  For display
argInit( &cparam, 1.5, 0, 0, 0, 0 );

The second parameter means zoom ratio for display
image size related to input image.

Graphics handling: libARgsub
  Set up and clean up the graphics window
void argInit( ARParam *cparam, double zoom,
int fullFlag, int xwin, int ywin,
int hmd_flag );
void argCleanup( void );

cparam: camera parameter
zoom: zoom ratio
fullFlag: 0: normal, 1: full screen mode
Xwin, ywin: create small window for debug
hmd_flag: 0: normal, 1: optical see-through mode

Sample1.c Main Function
main()!
{!
!init();!
!argMainLoop( mouseEvent, !!
!keyEvent, mainLoop); !
}!

  Go into the iterative cycle
void argMainLoop(
void (*mouseFunc)(int btn,int state,int x,int y),
void (*keyFunc)(unsigned char key, int x, int y),
void (*mainFunc)(void)
);

  Swap buffers
void argSwapBuffers( void );

Sample1.c - mainLoop Function
if( dataPtr = (ARUint8 *)
arVideoGetImage()) == NULL ) {
arUtilSleep(2);
return;
}
argDrawMode2D();
argDispImage(dataPtr, 0, 0 );
arVideoCapNext();
argSwapBuffers();

Image capture: libARvideo
  Return the pointer for captured image
  ARUint8 *arVideoGetImage( void );
  Pixel format and byte size are defined in config.h
  #define AR_PIX_FORMAT_BGR
  #define AR_PIX_SIZE 3

  Set the window for 2D drawing
void argDrawMode2D( void );
  Set the window for 3D drawing
void argDrawMode3D( void );
void argDraw3dCamera( int xwin, int ywin );
  Display image
void argDispImage( ARUint8 *image,
int xwin, int ywin );

Ex. 2: Detecting a Marker
  Key points
  Threshold value
  Important external variables
  arDebug – keep thresholded image
  arImage – pointer for thresholded image
  arImageProcMode – use 50% image for image
processing
-  AR_IMAGE_PROC_IN_FULL
-  AR_IMAGE_PROC_IN_HALF

Sample2.c – marker detection
/* detect the markers in the video frame */
if(arDetectMarker(dataPtr, thresh,
&marker_info, &marker_num) < 0 ) {
cleanup();
exit(0);
}
for( i = 0; i < marker_num; i++ ) {
argDrawSquare(marker_info[i].vertex,0,0);
}

Sample2.c – marker_info structure
typedef struct {
int area;
int id;
int dir;
double cf;
double pos[2];
double line[4][3];
double vertex[4][2];
} ARMarkerInfo; !

Ex. 3: Using a Pattern
  Key points
  Pattern files loading
  Structure of marker
information
-  Region features
-  Pattern Id, direction
-  Certainty factor
  Marker identification

Making a pattern template
  Use of utility program:
mk_patt.exe
  Show the pattern
  Put the corner of red line
segments on the left-top
vertex of the marker
  Pattern stored as a
template in a file
  1:2:1 ratio determines the
pattern region used

Sample3.c – Pattern File Loading
int patt_id;
char *patt_name = “Data/kanjiPatt”

/* load pattern file */
if(patt_id=arLoadPatt (patt_name) < 0)
{
printf ("Pattern file load error !! n");
exit(0);
}

Checking for known patterns
/* check for known patterns */
k = -1;
for( i = 0; i < marker_num; i++ ) {
if( marker_info[i].id == patt_id) {
/* you've found a pattern */
printf("Found pattern: %d n",patt_id);
if( k == -1 ) k = i;
else
/* make sure you have the best pattern
(highest confidence factor) */
if( marker_info[k].cf < marker_info[i].cf )
k = i;
}
}

Ex. 4 – Getting 3D information
  Key points
  Definition of a real marker
  Transformation matrix
-  Rotation component
-  Translation component

Sample4.c – Transformation matrix
double marker_center[2] = {0.0, 0.0};
double marker_width = 80.0;
double marker_trans[3][4];

arGetTransMat(&marker_info[i],
marker_center, marker_width,
marker_trans);

Finding the Camera Position
This function sets transformation matrix from marker
to camera into marker_trans[3][4]."
arGetTransMat(&marker_info[k], marker_center,
marker_width, marker_trans);

You can see the position information in the values of
marker_trans[3][4]."
" Xpos = marker_trans[0][3];
Ypos = marker_trans[1][3];
Zpos = marker_trans[2][3];

Ex. 5- Virtual Object Display
  Key points
  OpenGL parameter setting
  Setup of projection matrix
  Setup of modelview matrix

Appending your own OpenGL code
Set the camera parameters to OpenGL Projection matrix.
argDrawMode3D();
argDraw3dCamera( 0, 0 );
Set the transformation matrix from the marker to the camera to
the OpenGL ModelView matrix.
argConvGlpara(marker_trans, gl_para);
glMatrixMode(GL_MODELVIEW);
glLoadMatrixd( gl_para );

After calling these functions, your OpenGL objects are
drawn in the real marker coordinates.

3D CG Model Rendering
  ARToolKit does not have a function to handle
3D CG models.
  3rd party CG rendering software should be
employed.
  OpenVRML
  OpenSceneGraph
  etc

Loading Multiple Patterns
  Sample File: LoadMulti.c
  Uses object.c to load
  Object Structure
typedef struct {
char name[256];
int id;
int visible;
double marker_coord[4][2];
double trans[3][4];
double marker_width;
double marker_center[2];
} ObjectData_T;

Finding Multiple Transforms
  Create object list
ObjectData_T *object;

  Read in objects - in init( )
read_ObjData( char *name, int *objectnum );

  Find Transform – in mainLoop( )
for( i = 0; i < objectnum; i++ ) {
..Check patterns
..Find transforms for each marker
}

Drawing Multiple Objects
  Send the object list to the draw function
draw( object, objectnum );
  Draw each object individually
for( i = 0; i < objectnum; i++ ) {
if( object[i].visible == 0 ) continue;
argConvGlpara(object[i].trans, gl_para);
draw_object( object[i].id, gl_para);
}

Limitations of ARToolKit
  Partial occlusions cause tracking failure
  Affected by lighting and shadows
  Tracking range depends on marker size
  Performance depends on number of markers
  cf artTag, ARToolKitPlus
  Pose accuracy depends on distance to marker
  Pose accuracy depends on angle to marker

ARToolKit in the World

  Hundreds of projects
  Large research community

FLARToolKit
  Flash AS3 Version of the ARToolKit
(was ported from NyARToolkit the Java Version of the ARToolkit)

  enables augmented reality in the Browser
  uses Papervision3D for as 3D Engine
  available at http://saqoosha.net/
  dual license, GPL and commercial license

AR Application Components
Adobe Flash

Papervision 3D

FLARToolkit

private function mainEnter(e:Event):void {
/* Capture video frame*/
capture.draw(vid);

/* Detect marker */
if (detector.detectMarkerLite(raster, 80) && detector.getConfidence() > 0.5)
{
//Get the transfomration matrix for the current marker position
detector.getTransformMatrix(trans);

//Translates and rotates the mainContainer so it looks right
mainContainer.setTransformMatrix(trans);

//Render the papervision scene
renderer.render();
}
}

Papervision 3D
  http://www.papervision3d.org/
  Flash-based 3D-Engine
  Supports
  import of 3D Models
  texturing
  animation
  scene graph
  alternatives: Away3d, Sandy,…

Source Packages
  „Original“ FLARToolkit (Libspark, Saqoosha) (
http://www.libspark.org/svn/as3/FLARToolKit/trunk/ )

  Start-up-guides
  Saqoosha (http://saqoosha.net/en/flartoolkit/start-up-guide/ )
  Miko Haapoja (http://www.mikkoh.com/blog/?p=182 )
  „Frameworks“
  Squidder MultipleMarker – Example (
http://www.squidder.com/2009/03/06/flar-how-to-multiple-instances-of-multiple-
markers/ )
  FLARManager (http://words.transmote.com/wp/flarmanager/ )

Other Languages
  NyARToolKit
  http://nyatla.jp/nyartoolkit/wp/
  AS3, Java, C#, Processing, Unity, etc
  openFrameworks
  http://www.openframeworks.cc/
  https://sites.google.com/site/ofauckland/examples/8-artoolkit-example
  Support for other libraries
-  Kinect, Audio, Physics, etc

void testApp::update(){ //capture video and detect markers
mov.update();
if (mov.isFrameNew()) {
img.setFromPixels(mov.getPixels(), ofGetWidth(), ofGetHeight());
gray = img;
tracker.setFromPixels(gray.getPixels());
}
}
//--------------------------------------------------------------
void testApp::draw(){ //draw AR objects
ofSetColor(0xffffff); mov.draw(0, 0);
for (int i=0; i<tracker.markers.size(); i++) {
ARMarkerInfo &m = tracker.markers[i];
tracker.loadMarkerModelViewMatrix(m);
ofSetColor(255, 255, 0, 100); ofCircle(0,0,25); ofSetColor(0);
ofDrawBitmapString(ofToString(m.id),0,0);
}
}

Building Compelling AR Experiences

experiences

applications

tools Authoring

components Tracking, Display

Sony CSL © 2004

AR Authoring
  Software Libraries
  osgART, Studierstube, MXRToolKit
  Plug-ins to existing software
  DART (Macromedia Director), mARx, Unity,
  Stand Alone
  AMIRE, BuildAR, etc
  Next Generation
  iaTAR (Tangible AR)

mARx Plug-in

  3D Studio Max Plug-in
  Can model and view AR content at the same time

BuildAR

  http://www.buildar.co.nz/
  Stand alone application
  Visual interface for AR model viewing application
  Enables non-programmers to build AR scenes

Plug-ins for 3D authoring tools

  Unity - Game development tool (www.unity3d.com)
  Esperient Creator – General interactive 3D authoring
(www.esperient.com)

Metaio Design
  Complete commercial authoring platform
  http://www.metaio.com/software/design/
  Offers viewer and editor tools
  Drag and drop tools
  3D model import

Total Immersion D’Fusion Studio
  Complete commercial authoring platform
  http://www.t-immersion.com/
  Multi-platform
  Markerless tracking
  Scripting
  Face tracking
  Finger tracking
  Kinect support

Others
  AR-Media
  http://www.inglobetechnologies.com/
  Google sketch-up plug-in
  LinceoVR
  http://linceovr.seac02.it/
  AR/VR authoring package
  Libraries
  JARToolKit, MXRToolKit, ARLib, Goblin XNA

OSGART Programming Library
  Integration of ARToolKit with a High-Level
Rendering Engine (OpenSceneGraph)
OSGART= OpenSceneGraph + ARToolKit

  Supporting Geometric + Photometric Registration

osgART:Features

  C++ (but also Python, Lua, etc).
  Multiple Video Input supports:
  Direct (Firewire/USB Camera), Files, Network by
ARvideo, PtGrey, CVCam, VideoWrapper, etc.
  Benefits of Open Scene Graph
  Rendering Engine, Plug-ins, etc

Advanced Authoring: iaTAR (Lee 2004)

  Immersive AR Authoring
  Using real objects to create AR applications

More Information
•  Mark Billinghurst

–  mark.billinghurst@hitlabnz.org

•  Websites

–  www.hitlabnz.org

426 lecture 4: AR Developer Tools

More Related Content

What's hot (13)

Similar to 426 lecture 4: AR Developer Tools (20)

More from Mark Billinghurst (20)

Recently uploaded (20)

426 lecture 4: AR Developer Tools