The RoboCV Workshop

and
(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa
| Ajusal Sugathan & Utkarsh Sinha

presents

The word robot originally was supposed to
mean a slave
It is a machine which performs a variety
of tasks, either using manual external
control or intelligent automation
A manually controlled car or a ASIMOV
trying to kick a football are all robots
(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa | Ajusal Sugathan & Utkarsh Sinha

ž Robotics is a multi disciplinary field of
engineering encompassing the vistas of
› Mechanical design
› Electronic control
› Artificial Intelligence
ž It finds it’s uses in all aspects of our life
› automated vacuum cleaner
› Exploring the ‘Red’ planet
› Setting up a human colony there :D

ROBOTS
CONTROL
AUTONOMOUS
MANUAL
APPLICATIONS
INDUSTRIAL
MEDICAL
INTERFACE
HARDWARE
SOFTWARE
INTERLINKED

Ø Locomotion System
Ø Actuators
Ø Power Supply System
Ø Transmission System
Ø Switches
Ø Sensory Devices For Feedback
Ø Sensor Data Processing Unit

Ø A mobile robot must have a system to
make it move. Ob.
Ø This system gives our machine the ability
to move forward, backward and take
turns
Ø It may also provide for climbing up and
down
Ø Or even flying or floating J

Ø Each type of locomotion requires
different number of degrees of freedom
Ø More degrees of freedom means more the
number of actuators you will have to use
Ø Although one actuator can be used to
control more than one degree of freedom

Ø Wheeled
Ø Legged
Ø Climbing
Ø Flying
Ø Floating
Ø Snake-Like

Ø The kind of locomotion most frequently
used in robotics at the undergrad level
Ø This involves conversion of electrical
energy into mechanical energy (mostly
using motors)
Ø The issue is to control these motors to
give the required speed and torque

Ø We have a simple equation for the
constant power delivered to the motor:
› P = ζ X ω
Ø Note that the torque and angular velocity
are inversely proportionally to each other
Ø So to increase the speed we have to
reduce the torque

Ø The dc motors available have very high
speed of rotation which is generally not
needed
Ø At high speeds, they lack torque
Ø For reduction in speed and increase in
“pulling capacity” we use pulley or gear
systems

Ø Differential Drive
Ø Dual Differential Drive
Ø Car-type Drive
Ø Skid-steer Drive
Ø Synchronous Drive

Ø Simplest, easiest to implement and most
widely used.
Ø It has a free moving wheel in the front
accompanied with a left and right wheel. The
two wheels are separately powered
Ø When the wheels move in the same direction
the machine moves in that direction.
Ø Turning is achieved by making the wheels
oppose each other’s motion, thus generating
a couple

Ø In-place (zero turning radius) rotation is
done by turning the drive wheels at the same
rate in the opposite direction
Ø Arbitrary motion paths can be implemented
by dynamically modifying the angular
velocity and/or direction of the drive wheels
Ø Total of two motors are required, both of
them are responsible for translation and
rotational motion

Ø Simplicity and ease of use makes it the most
preferred system by beginners
Ø Independent drives makes it difficult for
straight line motion. The differences in
motors and frictional profile of the two
wheels cause them to move with slight
turning effect
Ø The above drawback must be countered with
appropriate feedback system. Suitable for
human controlled remote robots

Ø Uses synchronous rotation of its wheels to
achieve motion & turns
Ø It is made up of a system of 2 motors.
One which drive the wheels and the other
turns the wheels in a synchronous fashion
Ø The two can be directly mechanically
coupled as they always move in the same
direction with same speed

The direction of motion is given by
black arrow. The alignment of the
machine is shown by red arrow

Ø The use of separate motors for translation
and wheel turning guarantees straight
line motion without the need for dynamic
feedback control
Ø This system is somewhat complex in
designing but further use is much simpler

Ø Actuators, also known as drives, are
mechanisms for getting robots to move.
Ø Most actuators are powered by
pneumatics
(air pressure), hydraulics (fluid pressure),
or motors (electric current).
Ø They are devices which transform an
input signal (mainly an electrical signal))
into motion

Ø Widely used because of their
small size and high energy output.
Ø Operating voltage: usually 6,12,24V.
Ø Speed: 1-20,000 rpm..
Ø Power: P = ζ X ω

ØThe stator is the
stationary outside part of a
motor.
Ø The rotor is the inner
part which rotates.
Ø Red represents a
magnet or winding with a
north polarization.
Ø Green represents a
magnet or winding with a
south polarization.
Ø Opposite, red and
green, polarities attract.
Ø Commutator contacts
are brown and the brushes
are dark grey.

Ø Stator is composed of two or more
permanent magnet pole pieces.
Ø Rotor composed of windings which are
connected to a mechanical commutator.
Ø The opposite polarities of the energized
winding and the stator magnet attract and the
rotor will rotate until it is aligned with the
stator.
Ø Just as the rotor reaches alignment, the
brushes move across the commutator contacts
and energize the next winding.
Ø A yellow spark shows when the brushes
switch to the next winding.

ØIt is an electric
motor that can divide a
full rotation into a large
number of steps.
Ø The motor's position
can be controlled
precisely, without any
feedback mechanism.
Ø There are three
types:
Ø Permanent
Magnet
Ø Variable
Resistance
Ø Hybrid type

Ø Stepper motors work in a similar way to
dc motors, but where dc motors have 1
electromagnetic coil to produce movement,
stepper motors contain many.
Ø Stepper motors are controlled by
turning each coil on and off in a sequence.
Ø Every time a new coil is energized, the
motor rotates a few degrees, called the
step angle.

Full Step
Ø Stepper motors have 200
rotor teeth, or 200 full steps
per revolution of the motor
shaft.
Ø Dividing the 200 steps
into the 360º's rotation
equals a 1.8º full step angle.
Ø Achieved by energizing
both windings while
reversing the current
alternately.

ØServos operate on the principle of negative
feedback, where the control input is compared to
the actual position of the mechanical system as
measured.
ØAny difference between the actual and wanted
values (an "error signal") is amplified and used to
drive the system in the direction necessary to
reduce or eliminate the error
ØTheir precision movement makes them ideal for
powering legs, controlling rack and pinion steering,
to move a sensor around etc.

Ø Suitable power source is needed to run
the robots
Ø Mobile robots are most suitably powered
by batteries
Ø The weight and energy capacity of the
batteries may become the determinative
factor of its performance

Ø For a manually controlled robot, you can
use batteries or voltage eliminators
(convert the normal 220V supply to the
required DC voltage 12V , 24V etc.)

Ø Gear
Ø Belt Pulley
Ø Chain Sprocket
Ø Rack and Pinion
Ø Pick Place Mechanisms

Ø Gears are the most common means of
transmitting power in mechanical
engineering
Ø Gears form vital elements of mechanisms in
many machines such as vehicles, metal
tooling machine tools, rolling mills, hoisting
etc.
Ø In robotics its vital to control actuator speeds
and in exercising different degrees of
freedom

Ø To achieve torque magnification and
speed reduction
Ø They are analogous to transformers in
electrical systems
Ø It follows the basic equation:
Ø ω1 x r1 = ω2 x r2
Ø Gears are very useful in transferring
motion between different dimension

Ø An arrangement of gears to convert
rotational torque to linear motion
Ø Same mechanism used to steer wheels
using a steering
Ø In robotics used extensively in clamping
systems

Ø It allows for mechanical power, torque,
and speed to be transmitted across axes
Ø If the pulleys are of differing diameters, it
gives a mechanical advantage
Ø In robotics it can be used in lifting loads
or speed reduction
Ø Also it can be used in a differential drive
to interconnect wheels

Ø Sprocket is a profiled wheel with teeth that
meshes with a chain
Ø It is similar to the system found in bicycles
Ø It can transfer rotary motion between shafts
in cases where gears are unsuitable
Ø Can be used over a larger distance
Ø Compared to pulleys has lesser slippage due
to firm meshing between the chain and
sprocket

Ø For picking and placing many mechanisms
can be used:
vHook and pick
vClamp and pick
vSlide a sheet below and pick
vMany other ways
vLots of Scope for innovation

ž Image Processing is a tool for
analyzing image data in all areas of
natural science
ž It is concerned with extracting data
from real-world images
ž Differences from computer graphics is
that computer graphics makes
extensive use of primitives like lines,
triangles & points. However no such
primitives exist in a real world images.

ž Increasing need to replicate human
sensory organs
ž Eye (Vision) : The most useful and
complex sensory organ

ž Automated visual inspection system
Checking of objects for defects visually
ž Remote Sensing
ž Satellite Image Processing
ž Classification (OCR), identification
(Handwriting, finger prints) etc.
ž Detection and Recognition systems
(Facial recognition..etc)
ž Biomedical applications

ž Camera, Scanner or any other image
acquisition device
ž PC or Workstation or Digital Signal
Processor for processing
ž Software to run on the hardware platform
(Matlab, Open CV etc.)
ž Image representation to process the
image (usually matrix) and provide spatial
relationship
ž A particular color space is used to
represent the image(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa | Ajusal Sugathan & Utkarsh Sinha

Image Acquisition Device
(Eg. CCD or CMOS Camera)
Image Processor
(Eg. PC or DSP)
Image Analysis Tool
(Eg. Matlab or Open CV)
Machine Control Of Hardware
through serial or parallel interfacing

ž Using a camera
ž Analog cameras
ž Digital cameras
› CCD and CMOS cameras
ž Captures data from a single
light receptor at a time
ž CCD – Charge Coupled Devices
ž CMOS – Complementary MOSFET Sensor
based

ž Digital Cameras
› CCD Cameras
– High quality, low noise images
– Genarates analog signal converted using ADC
– Consumes high power
› CMOS Cameras
– Lesser sensitivity
– Poor image quality
– Lesser power
ž Analogue cameras require grabbing card
or TV tuner card to interface with a PC

Colored pixels on CCD
Chip

ž Matlab
ž Open CV

ž Two types: Vector and Raster
ž Vector images store curve information
ž Example: India’s flag
ž Three rectangles, one circle and
the spokes
ž We will not deal with vector images at
all

ž Raster images are different
ž They are made up of several dots

ž If you think about it, your laptop’s
display is a raster display
ž Also, vector images are high level
abstractions
ž Vector representations are more
complex and used for specific purposes

ž Raster
› Matrix
ž Vector
› Quadtrees
› Chains
› Pyramid
Of the four, matrix is the most general.
The other three are used for special
purposes. All these representations must
provide for spatial relationships

ž Computers cannot handle continuous
images but only arrays of digital
numbers
ž So images are represented as 2-D
arrays of points (2-D matrix)(Raster
Represenatation)
ž A point on this 2-D grid (corresponding
to the image matrix element) is called
PIXEL (picture element)
ž It represents the average irradiance
over the area of the pixel

ž Each pixel requires some memory
ž Color depth : Amount of memory each
pixel requires
ž Examples
› 1-bit
› 8-bit
› 32-bit
› 64-bit

ž Pixels are tiny little dots of color you see on
your screen, and the smallest possible size
any image can get
ž When an image is stored, the image file
contains information on every single pixel in
that image i.e
› Pixel Location
› Intensity
ž The number of pixels used to represent the
image digitally is called Resolution
ž More the number of pixels used, higher the
resolution
ž Higher resolution requires more
processing power

ž MATLAB stands for MATrix LABoratory,
a software developed by Mathworks
Inc (www.mathworks.com). MATLAB
provides extensive library support for
various domains of scientific and
engineering computations and
simulations

ž When you click the MATLAB icon (from
your desktop or Start>All Programs),
you typically see three windows:
Command Window, Workspace and
Command History. Snapshots of these
windows are shown below

ž This window shows the variables
defined by you in current session on
MATLAB

ž Command History stores the list of
recently used commands for quick
reference

ž This is where you run your code

ž In MATLAB, variables are stored as
matrices (singular: matrix), which
could be either an integer, real
numbers or even complex numbers
ž These matrices bear some
resemblance to array data structures
(used in computer programming)

ž Let us start with writing simple
instructions on MATLAB command
window
ž To define an integer,
ž Type a=4 and hit enter
ž >>a=4
ž To avoid seeing the variable, add a
semicolon after the instruction
ž >>a=4;

ž Similarly to define a 2x2 matrix, the
instruction in MATLAB is written as
ž >> b=[ 1 2; 3 4];
ž If you are familiar with operations on
matrix, you can find the determinant
or the inverse of the matrix.
ž >> determin= det(b)
ž >> d=inv(b)

ž Images as we have already seen are
stored as matrices
ž So now we try to see this for real on
MATLAB
ž We shall also look into the basic
commands provided by MATLAB’s
Image Processing Toolbox

ž Once you have started MATLAB, type the
following in the Command Window
ž >> im=imread(‘sample.jpg');
ž This command stores the file image file
‘sample.jpg’ in a variable called ‘im’
ž It takes this file from the Current-
Directory specified
ž Else, entire path of file should be
mentioned

ž You can display the image in another
window by using imshow command
ž >>figure,imshow(im);
ž This pops up another window (called as
figure window), and displays the image
‘im’

ž The ‘imview’ command can also be
used in order toview the image
ž imview(im);
ž Difference is that in this case you can
see specific pixel values just by moving
the cursor over the image

ž To know the breadth and height of the
image, use the size function,
ž >>s=size(im);
ž The size function basically gives the
size of any array in MATLAB
ž Here we get the size of the IMAGE
ARRAY

ž Now that we have our image stored in
a variable we can observe and
understand the following:
ž How pixels are stored?
ž What does the values given by each
pixel indicate?
ž What is Image Resolution?

ž Have a look at the values stored
ž Say the first block of 10 x 10
ž >>im(1:10,1:10);
ž Or Say view the pixel range 50:150 on
both axis
ž >>
figure,imshow(im(50:150,50:150));

ž 1-bit = BLACK or WHITE
ž 8-bit = 28 different shades
ž 24-bit = 224 different shades
ž 64-bit images – High end displays
ž Used in HDRI, storing extra
information per pixel, etc

ž This is another name for 1-bit images
ž Each pixel is either White or Black
ž Technically, this is a black & white
image

ž Another name for 8-bit images
ž Each pixel can be one of 256 different
shades of gray
ž These images are popularly called
Black & White. Though, this is
technically wrong.

ž Again, each pixel gets 8 bits
ž But each of the 256 values maps to a
color in a predefined “palette”
ž If required, you can have different
bit depths

ž We won’t be dealing with indexed
images

ž 8-bits is too less for all the different
shades of colors we see
ž So 24-bits is generally used for color
images
ž Thus each pixel can have one of 224
unique colors

ž Now, a new problem arises:
ž How do you manage so many different
shades?
ž Programmers would go nuts
ž Then came along the idea of color
spaces

ž A color space can be thought of as a
way to manage millions of colors
ž Eliminates memorization, and
increases predictability
ž Common color spaces:
› RGB
› HSV
› YCrCb or YUV
› YIQ

ž You’ve probably used this already

ž Each pixel stores 3 bytes of data
ž The 24-bits are divided into three 8-bit
values
ž The three are: Red, Green and Blue i.e
the primary colours
ž Mixing of primary colours in right
proportions gives any particular colour
ž Each pixel has these 3 values

ž 1 byte = 8 bits can store a value
between 0-255
ž We get pixel data in the form RGB
values with each varying from 0-255
ž That is how displays work
ž So there are 3 grayscale channels

ž Advantages:
› Intuitive
› Very widely used
ž Disadvantages:
› Image processing is relatively tough

ž HSV makes image processing easier
ž Again, 24 bits = three 8-bit values or 3
channels
ž The 3 channels are:
› Hue
› Saturation (Shade of Colour)
› Value (Intensity)

ž The Hue is the tint of color used
› It represents the colour of the pixel (Eg. Red
Green Yellow etc)
ž The Saturation is the “amount” of
that tint
› It represents the intensity of the colour (Eg.
Dark red and light red)
ž The Value is the “intensity” of that
pixel
› It represents the intensity of brightness of the
colour

ž RGB image converted to HSV
RGB
HUE
SATURATION
VALUE

ž Advantages:
› The color at a pixel depends on a single
value
› Illumination independent
ž Disadvantages:
› Something

ž Intuitively RGB might seem to be the simpler
and better colour space to deal with
ž Though HSV has its own advantages
especially in colour thresholding
ž As the colour at each pixel depends on a
single hue value it is very useful in
separating out blobs of specific colours even
when there are huge light variations
ž Thus it is very useful in processing real
images taken from camera as there is a large
amount of intensity variation in this case
ž Hence, ideal for robotics applications

ž Widely used in digital video
ž Has three 8-bit channels:
› Y Component:
– Gives luminance or intensity
› Cr Component:
– It is the RED component minus a reference value
› Cb Component:
– It is the BLUE component minus a reference
value
ž Hence Cr and Cb components represent
the colour called “Color Difference
Components”

ž Advantages:
› Used in video processing
› Gives you a 2-D colour space hence helps
in closer distinguishing of colours
ž Disadvantages:

ž The camera returns images in a certain
color space
ž You might want to convert to different
color spaces to process it
ž Colour space conversions can take
place between RGB to any other colour
space and vice versa

ž Since cameras usually input images in
rgb
ž We would like to convert these images
into HSV or YCrCb
ž Conversions:
› RGB->HSV
› HSV->RGB
› RGB->YCrCb
› YCrCb->RGB

ž RGB -> HSV

(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa |
Ajusal Sugathan & Utkarsh Sinha
ž HSV RGB YCrCb

ž >>h = rgb2hsv(im)
ž This converts the RGB image to HSV
ž The new colour space components can be
seen using
ž >> imview(h)
ž >> imview(h(:,:,1)) “—HUE—”
ž >> imview(h(:,:,2)) “—Saturation—
”
ž >> imview(h(:,:,3)) “—Value—”

ž >>R = hsv2rgb(im)
ž This converts the HSV image to RGB
ž The new colour space components can
be seen using
ž >> imview(R)
ž >> imview(R(:,:,1)) “—Red—”
ž >> imview(R(:,:,2)) “—Green—”
ž >> imview(R(:,:,3)) “—Blue—”

ž >> Y = rgb2ycbcr(im);
ž This converts the RGB image to YCbCr
ž The new colour space components can be
seen using
ž >> imview(Y)
ž >> imview(Y(:,:,1)) “—Luminance—”
ž >> imview(Y(:,:,2)) “—Differenced
Blue—”
ž >> imview(Y(:,:,3)) “—Differenced
Red—”

ž >> R = ycbcr2rgb(im);
ž This converts the YCbCr image to RGB
ž The new colour space components can
be seen using
ž >> imview(R)
ž >> imview(R(:,:,1)) “—Red—”
ž >> imview(R(:,:,2)) “—Green—”
ž >> imview(R(:,:,3)) “—Blue—”

ž Formulae for conversion are very
complex
ž But the best thing is, you don’t need to
remember these formulae
ž Matlab and OpenCV have built-in
functions for these transformations :-)

ž OpenCV is a collection of many
functions that help in image processing
ž You can use OpenCV in C/C++, .net
languages, Java, Python, etc as well
ž We will only discuss OpenCV in C/C++

ž It is blazingly fast
ž Quite simple to use and learn
ž Has functions for machine learning,
image processing, and GUI creation

ž Download the latest OpenCV package
from
http://sourceforge.net/projects/opencv
/
ž Install the package, and note where
you installed it (like C:Program
FilesOpenCV)

ž Now, we need to tell Microsoft Visual
Studio that we’ve installed OpenCV
ž So, we tell it where to find the OpenCV
header files
ž Start Microsoft Visual Studio 2008

1
2

Type
these
paths
into the
list

ž Right now, Visual Studio knows where
to find the OpenCV include files and
library files
ž Now we create a new project

ž Accept all default settings in the
project
ž You’ll end up with an empty project
with a single file (like Mybot.cpp)
ž Open this file, we’ll write some code
now

ž Add the following at the top of the
code
#include <cv.h>
#include <highgui.h>
ž This piece of code includes necessary
OpenCV functionality

ž Now, we get to the main() function
int main()
{
ž The main function is where for
program execution begins

ž Next, we load an image
IplImage* img = cvLoadImage("C:hello.jpg");
ž The IplImage is a data type, like int,
char, etc

ž Comes built-into OpenCV
ž Any image in OpenCV is stored as an
IplImage thingy
ž It is a “structure”

ž Opens filename and returns it as an
IplImage structure
ž Supported formats:
› Windows bitmaps - BMP, DIB
› JPEG files - JPEG, JPG, JPE
› Portable Network Graphics - PNG
› Portable image format - PBM, PGM, PPM
› Sun rasters - SR, RAS
› TIFF files - TIFF, TIF
› OpenEXR HDR images - EXR
› JPEG 2000 images - jp2

ž Now we show this image in a window
cvNamedWindow("myfirstwindow");
cvShowImage("myfirstwindow", img);
ž This uses some HighGUI functions
(comes along with OpenCV)

ž Creates a window with the caption title
ž This is a HighGUI function
ž You can add controls to each window
as well (track bars, buttons, etc)

ž Shows img in the window with caption
title
ž If no such window exists, nothing
happens

ž Finally, we wait for an input, release
and exit
cvWaitKey(0);
cvReleaseImage(&img);
return 0;
}

ž Waits for time milliseconds, and
returns whatever key is pressed
ž If time=0, waits till eternity
ž Here, we’ve used it to keep the
windows from vanishing immediately

ž Erases img from the RAM
ž Get rid of an image as soon as
possible. RAM is precious J
ž Note that you send the address of the
image (&img) and not just the image
(img)

ž Right now, Visual Studio knows where
OpenCV is
ž But it does not know, whether to use
OpenCV or not
ž We need to tell this explicitly

ž Got errors?
› Check if the syntax is correct
› Copy all DLL files in *OpenCVbin into
C:WindowsSystem32

ž src is the original image
ž dst is the destination
ž code is one of the follow:
› CV_BGR2HSV
› CV_RGB2HSV
› CV_RGB2YCrCb
› CV_HSV2RGB
› CV_<src_space>2<dst_space>

ž src should be a valid image. Or an
error will pop up
ž dst should be a valid image, i.e. you
need a blank image of the same size
ž code should be valid (check the
OpenCV documentation for that)

ž Allocates memory for an image of size
size, with bits bits/pixel and chan
number of channels
ž Used for creating a blank image
ž Use cvSize(width, height) to specify
the size

ž Example:
› IplImage* blankImg =
cvCreateImage(cvSize(640, 480), 8, 3);

ž Wired
› Motor Driving module
› Interface with PC (Parallel/Serial)
ž Wireless
› The Motor-driving module
› The Wireless Receiver Circuit
› The Wireless Transmitter Circuit
› Interface with PC (Parallel/Serial)

ž IC 7805 Voltage Regulator
ž L293D Motor Driver
ž MCT2E Opto-Coupler
ž Parallel Port Male-Connector
ž RF-RX Connector

ž It’s a three terminal linear 5 volt
regulator used to supply the board and
other peripherals
ž Prescribed input voltage to this
component is about 7-9 Volts

ž Voltage fluctuations can be controlled
by using low pass filter capacitors
across output and input
ž Higher input voltage can be applied if
heatsink is provided

ž Used to control Dc and Stepper Motors
ž Uses a H-Bridge which is an electronic
switching circuit that can reverse direction of
current
ž It’s a Dual-H bridge
ž Basically used to convert a low voltage input
into a high voltage output to drive the motor
or any other component
ž Eg: Microcontrollerà Motor Driverà
Motor
(5 Volts) (12
Volts)

ž Different Motor Driver ICs
› L293D
– 600mA Current Rating
– Dual H-bridge (Dc and Stepper Motors)
› L298N
– 1 Amp Current Rating
› L297-L298 (Coupled)
– For stepper motor overdriving
– 2 Ics in parallel
› ULN2003/ULN2803
– 500mA Current Rating
– For unipolar stepper motors

ž Output Current:
› 600 mA
ž Output Voltage
› Wide Range
› 4.5 V – 36 V

ž There are many situations where
signals and data need to be transferred
from one subsystem to another within
a piece of electronics
ž Relays are too bulky as they are
electromechanical in nature and at the
same time give lesser efficiency
ž In these cases an electronic
component called Optocoupler is used

ž They are generally used when the 2
subsystems are at largely different
voltages
ž These use a beam of light to transmit
the signals or data across an electrical
barrier, and achieve excellent isolation

ž In our circuit, Opto-isolator (MCT2E) is
used to ensure electrical isolation
between motors and the PC parallel port
during wired connection
ž The Viz-Board has four such chips to
isolate the four data lines (pin 2, pin 3,
pin 4, pin 5) coming out of the parallel
port

ž Along with the Viz-Board 2 extensions
have been provided i.e
› The Rf Transmitter Module
› The Rf Reciever Module
Transmitt
er
Receive
r

ž Radio frequency modules are used for
data transmission wirelessly at a certain
frequency
ž It sends and receives radio waves of a
particular frequency and a decoder and
encoder IC is provided to encode and
decode this information
ž Wireless transmission takes place at a
particular frequency Eg. 315Mhz
ž Theses modules might be single or dual
frequency

ž Antenna is recommended on both of
them - just connect any piece of 23 cm
long to the Antenna pin
ž The kit has a dual frequency RF
module with frequencies 315/434 Mhz

ž The encoder IC encodes the parallel
port data and sends it to the RF
transmitter module for wireless
transmission
ž They are capable of encoding
information which consists of N
address bits and (12-N) data bits

ž The HT12E Encoder IC has 8 address
bits and 4 data bits
ž A DIP-Switch can be used to set or
unset the address bits A0-A7

A0-A7—Address
Bits
AD8-AD11—Data
Bits

ž The decoder IC decodes the RF
transmitter data and sends it to the
parallel port for wireless transmission
ž They are capable of encoding
information which consists of N
address bits and (12-N) data bits

ž The HT12D Decoder IC has 8 address
bits and 4 data bits
ž A DIP-Switch can be used to set or
unset the address bits A0-A7

A0-A7—Address
Bits
D8-D11—Data
Bits

ž Serial Port
ž Parallel Port
ž USB

ž Data is transferred serially i.e packets
are sent one after the other through a
single port

ž Data is transferred in parallel through
different data pins at the same time
ž Communication is pretty fast
ž Found in old printer ports

25th pin : Ground
2nd-12th pin : I/O lines

ž Parallel port is faster than serial
ž A mass of data can be transmitted at
the same time through parallel ports
ž Though parallel and serial ports are not
found these days in laptops
ž Desktops and old laptops have these
ports

Direct Output from parallel port
Output from motor driver

Camera, object and source positions

Image sampling
and quantization

Continuous image
projected
on an array sensor
Result of image sampling
and quantization

Sampling:
Digitizing the coordinate
values
(spatial resolution)
Quantization:
Digitizing the amplitude
values
(intensity levels)

• 1 bit /pixel
• B bits/pixel
–2B gray levels
–1 byte = 8 bits –> 256
levels
–2 possible values
–2 gray levels -> 0 or 1 (binary
image)

ž All this sampling and quantization puts
in extra noise on the image!
ž Noise can be reduced by
› Using hardware
› Using software: filters

ž Why do we need to enhance images?
ž Why filter images?

ž Large amounts of external
disturbances in real images
ž Due to different factors like changing
lighting and other real-time effects
ž To improve quality of a captured image
to make it easier to process the image

ž First step in most IP applications
ž Used to remove noise in the input
image
ž To remove motion blur from an image
ž Enhancing the edges of an image to
make it appear sharper

ž Generally used types Of Filtering
› Averaging Filter
› Mean Filter
› Median Filter
› Gaussian Smoothing
› Histogram Equalization

ž The Averaging filter is used to sharpen
the images by taking average over a
number of images
ž It eliminates noise by assuming that
different snaps of the same image
have different noise patterns

ž Noise is gaussian in nature i.e follows a
gaussian curve
ž Hence, summing up noises infinite times
approaches zero

ž This is extremely useful for satellites
that take intergalactic photographs
ž The images are extremely faint, and
there is more noise than the image
itself
ž Millions of pictures are taken, and
averaged to get a clear picture

ž The Mean is used to soften an image
by averaging surrounding pixel values
Center pixel =
(22+77+48+150+77+158+0+77+219)/9

ž The center pixel would be changed
from 77 to 92 as that is the mean
value of all surrounding pixels
ž This filter is often used to smooth
images prior to processing
ž It can be used to reduce pixel flicker
due to overhead fluorescent lights

ž This replaces each pixel value by the
median of its neighbors, i.e. the value
such that 50% of the values in the
neighborhood are above, and 50% are
below
ž This can be difficult and costly to
implement due to the need for sorting
of the values
ž However, this method is generally very
good at preserving edges

ž Its performance is particularly good for
removing short noise
ž The median is calculated by first sorting all
the pixel values from the surrounding
neighborhood into numerical order and then
replacing the pixel being considered with the
middle pixel value
ž If the neighborhood under consideration
contains an even number of pixels, the
average of the two middle pixel values is
used

ž Used to `blur' images and remove
detail and noise
ž The effect of Gaussian smoothing is to
blur an image
ž The Gaussian outputs a `weighted
average' of each pixel's neighborhood,
with the average weighted more
towards the value of the central pixels

ž A Gaussian provides gentler smoothing
and preserves edges better than a
similarly sized mean filter
Before
Blurring
After
Blurring

ž It is very useful in contrast enhancement
ž Especially to eliminate noise due to
changing lighting conditions etc
ž Transforms the values in an intensity
image so that the histogram of the output
image approximately matches a specified
histogram

Filters and histograms

ž ‘Imfilter’ function is used for creating
different kinds of filters In MATLAB
ž B = imfilter(A,H,’option’) filters the
multidimensional array A with the
multidimensional filter H
ž The array A can be a nonsparse numeric
array of any class and dimension
ž The result B has the same size and class as A

ž Options in imfilter
ž Convolution is same as correlation except
that the h matrix is inverted before applying
the filter

ž h = ones(5,5) / 25;
ž imsmooth = imfilter(im,h);
ž Here a mean filter is implemented using
the appropriate ‘h’ matrix
ž imshow(im), title('Original Image');
ž figure, imshow(imsmooth),
title('Filtered Image')

ž FSPECIAL is used to create predefined
filters
ž h = FSPECIAL(TYPE);
ž FSPECIAL returns h as a computational
molecule, which is the appropriate
form to use with imfilter

ž The process of adjusting intensity
values can be done automatically by
the histeq function
ž >>im = imread('pout.tif');
ž >>jm = histeq(im);
ž >>imshow(jm)
ž >>figure, imhist(jm,64)

Original Image

Histogram Equalized Image

ž Things aren’t as simple as they were in
Matlab
ž C/C++ needs a bit of syntax and
formalities

ž We’ll try doing the following right now
› Gaussian filter
› Median filter
› Bilateral filter
› Simple blur
› Averaging filter

ž Start Microsoft Visual Studio 2008
ž I assume you have OpenCV installed

#include <cv.h>

#include <cv.h>
int main()
{

#include <cv.h>
int main()
{
IplImage* img = cvLoadImage(“C:noisy.jpg”);

#include <cv.h>
int main()
{
IplImage* img = cvLoadImage(“C:noisy.jpg”);
IplImage* imgBlur = cvCreateImage(cvGetSize(img),
8, 3);
IplImage* imgGaussian = cvCreateImage(cvGetSize
(img), 8, 3);
IplImage* imgMedian = cvCreateImage(cvGetSize
(img), 8, 3);
IplImage* imgBilateral = cvCreateImage(cvGetSize
(img), 8, 3);

cvSmooth(img, imgBlur, CV_BLUR, 3, 3);
cvSmooth(img, imgGaussian, CV_GAUSSIAN, 3, 3);
cvSmooth(img, imgMedian, CV_MEDIAN, 3, 3);
cvSmooth(img, imgBilateral, CV_BILATERAL, 3, 3);

cvNamedWindow(“original”);
cvNamedWindow(“blur”);
cvNamedWindow(“gaussian”);
cvNamedWindow(“median”);
cvNamedWindow(“bilateral”);
cvShowImage(“original”, img);
cvShowImage(“blur”, imgBlur);
cvShowImage(“gaussian”, imgGaussian);
cvShowImage(“median”, imgMedian);
cvShowImage(“bilateral”, imgBilateral);
cvWaitKey(0);
return 0;
}

ž Blur: The plain simple Photoshop blur
ž Gaussian: The best result (preserved
edges and smoothed out noise)
ž Median: Nothing special
ž Bilateral: Got rid of some noise, but
preserved edges to a greater extend

ž Your OpenCV installation comes with
detailed documentation
ž *OpenCVdocsindex.html
ž Scroll down, and you’ll see OpenCV
Reference Manuals

ž Try looking up cvSmooth in the CV
Reference Manual

ž Now try looking up cvEqualizeHist

ž There are no built-in functions for this
ž So, we’ll code it ourselves
ž And this will be a good exercise for
getting better at OpenCV

#include <cv.h>
int main()
{
IplImage* imgRed[25];
IplImage* imgGreen[25];
IplImage* imgBlue[25];
Holds the R, G and B
channels separately
for each of the 25
images

IplImage* imgBlue[25];
for(int i=0;i<25;i++)
{
IplImage* img;
char filename[150];
sprintf(filename, "%d.jpg", (i+1));
img = cvLoadImage(filename);
• Generate the strings “1.jpg”,
“2.jpg”, etc and store them into
filename
• Load the image filename

img = cvLoadImage(filename);
imgRed[i] = cvCreateImage(cvGetSize(img), 8,
1);
imgGreen[i] = cvCreateImage(cvGetSize(img), 8,
1);
imgBlue[i] = cvCreateImage(cvGetSize(img), 8,
1);
• Allocate memory for each component
of image i

imgBlue[i] = cvCreateImage(cvGetSize(img), 8,
1);
cvSplit(img, imgBlue[i], imgGreen[i],
imgRed[i], NULL);
cvReleaseImage(&img);
}
• Split img into constituent channels
• Note the order: B G R
• Release img

ž We created 75 grayscale images: 25
for red, 25 for green and 25 for blues
ž Loaded 25 color images in the loop
ž Split each image, and stored in an
appropriate grayscale image

CvSize imgSize = cvGetSize(imgRed[0]);
IplImage* imgResultRed = cvCreateImage(imgSize, 8,
1);
IplImage* imgResultGreen = cvCreateImage(imgSize,
8, 1);
IplImage* imgResultBlue = cvCreateImage(imgSize,
8, 1);
IplImage* imgResult = cvCreateImage(imgSize, 8,
3);
• This will hold the final, filtered image
• It will be a combination of the
grayscale channels imgResultRed,
imgResultGreen and imgResultBlue

IplImage* imgResult = cvCreateImage(imgSize, 8,
3);
for(int y=0;y<imgSize.height;y++)
{
for(int x=0;x<imgSize.width;x++)
{
• Two loops to take us through the
entire image

for(int x=0;x<imgSize.width;x++)
{
int theSumRed=0;
int theSumGreen=0;
int theSumBlue=0;
{
• To figure out the average, we need to
find the numerator (the sum) over all
25 images

{
theSumRed+=cvGetReal2D(imgRed[i], y,
x);
theSumGreen+=cvGetReal2D(imgGreen[i],
y, x);
theSumBlue+=cvGetReal2D(imgBlue[i], y,
x);
}
• To figure out the average, we need to
find the numerator (the sum) over all
25 images

theSumRed = (float)theSumRed/25.0f;
theSumGreen = (float)theSumGreen/25.0f;
theSumBlue = (float)theSumBlue/25.0f;
cvSetReal2D(imgResultRed, y, x,
theSumRed);
cvSetReal2D(imgResultGreen, y, x,
theSumGreen);
cvSetReal2D(imgResultBlue, y, x,
theSumBlue);
}
}
• Once we have the sum, we divide by
25 and set the appropriate pixels

cvMerge(imgResultBlue, imgResultGreen,
imgResultRed, NULL, imgResult);
cvNamedWindow("averaged");
cvShowImage("averaged", imgResult);
cvWaitKey(0);
return 0;
}
• Merge the three channels, and
display the image

ž cvLoadImage always loads as BGR
ž cvSplit to get the individual channels
ž cvMerge to combine individual
channels into a color image

ž IplImage to store any image in
OpenCV
ž cvCreateImage to allocate memory
ž cvReleaseImage to erase an image
from the RAM

ž cvWaitKey to get a keypress within
certain milliseconds
ž cvNamedWindow to create a window
ž cvShowImage to show an image in a
window

ž cvGetReal2D to get value at a pixel in
grayscale images
ž cvSetReal2D to set the value at a pixel
ž CvSize to store an image’s size

ž you can always refer to the OpenCV
documentation

ž The process of extracting image
components that are useful in
representation of image for some
particular purpose
ž Basic morphological operations are:
› Dilation
› Erosion

ž The operation that grows or thickens
objects in a binary image
ž The specific manner of thickening is
controlled by a shape referred to as
“structuring element”

Structuring
Element
Binary
Image

Dilated
Image

ž Erosion shrink or thins objects in a
binary image
ž The manner of shrinkage is controlled
by the structuring element

Eroded
Image

ž In practical image processing dilation
and erosion are performed in various
combinations
ž An image can undergo a series for
diltions and erosion using the same or
different structuring element

ž In practical image processing dilation
and erosion are performed in various
combinations
ž An image can undergo a series for
diltions and erosion using the same or
different structuring element
ž Two Common Kinds:
› Morphological Opening
› Morphological Closing

ž It is basically one erosion followed by
one dilation by the same structuring
element
ž They are used to smooth object
contours, break thin connections and
remove thin protrusions

A—Image
B—Structuring element

ž It is basically one dilation followed by one
erosion by the same structuring element
ž They are used to smooth object contours
like opening
ž But unlike opening they generally join
narrow breaks, fill long thin gulfs and fills
holes smaller than the structuring
element

ž Used to generate a structuring
element
ž >>se=strel(shape,parameters)

ž Dilation in matlab is done using the
following command:
ž >>bw2=imdilate(bw,st)
ž Bw = Original image
ž St = Structuring element

ž Erosion in matlab is done using the
following command:
ž >>bw2=imerode(bw,st)

ž Opening in matlab is done using the
following command:
ž >>bw2=imopen(bw,st)

ž Closing in matlab is done using the
following command:
ž >>bw2=imclose(bw,st)

ž cvErode(src, dst)
ž cvDilate(src, dst)
ž Opening & closing: use the appropriate
sequence

ž By default, OpenCV uses the zero
structuring element (all are zeros)
ž You can explicitly specify your
structuring element as well
ž Check the OpenCV Documentation for
more information

ž Computers can manipulate images
very efficiently
ž But, comprehending an image with
millions of colors is tough
ž Solution: Figure out interesting
regions, and process them

ž Each pixel is checked for its value
ž If it lies within a range, it is marked as
“interesting” (or made white)
ž Otherwise, it’s made black
ž Figuring out the range depends on
lighting, color, texture, etc

ž Demo thresholdRGB
ž Demo thresholdHSV

ž MATLAB provides a facility to execute
multiple command statements with a
single command. This is done by
writing a .m file
ž Goto File > New > M-file
ž For example, the graythresh function
can be manually written as a m-file as:

ž Observe that, comments (in green) can
be written after the symbol ‘%’. A
commented statement is not considered
for execution
ž M-files become a very handy utility for
writing lengthy programs and can be
saved and edited, as and when required
ž We shall now see, how to define your own
functions in MATLAB.

ž Functions help in writing organized
code with minimum repetition of logic
ž Instead of rewriting the instruction set
every time, you can define a function
ž Syntax:
ž Create an m-file and the top most
statement of the file should be the
function header
ž function [return values] = function-
name(arguments)

ž The inbuilt graythresh function in
matlab is used for thresholding of
grayscale images
ž It uses the Otsu’s Method Of
thresholding
ž A sample thresholding opreation has been
shown in the next slide

Image thresholded for
the colour blue

The Real thing J

ž Thresholding of a grayscale image can
be done in MATLAB using the following
commands:
ž >> level=graythresh(imGRAY);
ž >> imBW = im2bw(imGRAY,level);
ž >> imview(imBW);

ž The graythresh command basically gives
an idea as to what exactly the threshold
value should be
ž Graythresh returns a value that lies in the
range 0-1
ž This gives the level of threshold which is
obtained by a complex method called the
Otsu’s Method of Thresholding

ž This level can be converted into pixel
value by multiplying by 255
ž Lets say, level=.4
ž Then threshold value for the grayscale
image is:
ž 0.4 x 255 =102

ž What this indicates is that for the given
image the values below 102 have to
be converted to 0 and values from
103-255 to the value 1
ž Conversion from grayscale to binary
image is done using the function:
ž >>imBW = im2bw(imGRAY,level);

ž Here level is the threshold level
obtained from graythresh function
ž This function converts pixel intensities
between 0 to level to zero intensity
(black) and between level+1 to 255 to
maximum (white)

ž In order to threshold an RGB colour
image using the graythresh function, the
following have to be done:
› Conversion of the RGB image into its 3
grayscale components
› Subtracting each of these components from
the other 2 to get the pure colour intensities
› Finding level for each of the grayscale using
graythresh
› Thresholding the image using imbw and the
level

ž Commands:
ž Im=Imread(‘rgb.jpg’);
ž R = im(:,:,1); --Red
ž G = im(:,:,2); --Green
ž B = im(:,:,3); --Blue

ž Ronly=R-B-G; --Pure RED
ž Gonly=G-R-B; --Pure GREEN
ž Bonly=B-G-R; --Pure BLUE

ž Levelr=graythresh(Ronly);
ž Levelg=graythresh(Gonly);
ž Levelb=graythresh(Bonly);

ž Rthresh=im2bw (Ronly,levelR);
ž Gthresh=im2bw(Gonly,levelG);
ž Bthresh=im2bw(Bonly,levelB);

ž Using a manually designed thresh_tool
function to adjust the levels as
required
ž To get a feel of how levels vary

s=size(im);
temp=im;
thresh=128;
for i=1:s(1,1)
for j=1:s(1,2)
if temp(i,j)<thresh
temp(i,j)=0;
else
temp(i,j)=255;
end
end
end
imview(temp);

ž Splitting of HSV image into
components
ž Using the Hue channel and
thresholding it for different values
ž Since the hue value of a single colour
is constant it is relatively simple to
threshold and gives better accuracy

function [temp] = ht(im,level1,level2)
s=size(im);
temp=im;
for i=1:s(1,1)
for j=1:s(1,2)
if (temp(i,j)<level2 & temp(i,j)>level1)
temp(i,j)=1;
else
temp(i,j)=0;
end
end
end
imview(temp);

ž To this function we give the input
arguments as the upper and lower
bounds of the threshold levels
ž These levels can be obtained by
having a look at the range of hue
values for the particular colour

Now that you know the
basics

ž cvThreshold(src, dst, threshold, max,
type)
ž type:
› CV_THRESH_BINARY
› CV_THRESH_BINARY_INV
› And several others (check documentation)

ž cvInRangeS(src, scalarLower,
scalarUpper, dst);
ž scalarLower = cvScalar(chan1, chan2,
chan3, chan4);
ž scalarUpper = cvScalar(chan1, chan2,
chan3, chan4);

ž Ultra basics: motors, drives, etc
ž Digital image representation
ž Color spaces
ž Inter-conversion of color spaces
ž Electronics
ž Filtering
ž Thresholding
ž Morphology

ž After thresholding, we get a binary
image
ž We want useable information like
centers, outlines, etc
ž There geometrical properties can be
found using many methods. We’ll talk
about moments and contours only.

ž Moments are a mathematical concept
ž ∑ ∑intensity*xxorder*yyorder

ž Consider xorder=0 and yorder=0 for a
binary image
ž So you’re just summing up pixel values
ž This means, you’re calculating the area
of the white pixels

ž Now consider xorder=1 and yorder=0
for a binary image
ž You sum only those x which are white
ž So you’re calculating the numerator of
an average

ž The number of points where the pixel
is white is the area of the image
ž So, dividing this particular moment
(xorder=1, yorder=0) by the earlier
example (xorder=0, yorder=0) gives
the average x
ž This is the x coordinate of the centroid
of the blob

ž Similarly, for xorder=0 and yorder=1,
you’ll get the y coordinate

ž The order of a moment =
xorder+yorder
ž So, the area is a zero order moment
ž The centroid coordinate = a first order
moment / the zero order moment

ž There are entire books written on this
topic
ž You can find complex geometrical
properties, like the eccentricity of an
ellipse, radius of curvature of objects,
etc
ž Also check for Hu invariants if you’re
interested

Centroid Area etc

ž These are pixels of an image that are
conencted to each other forming
separate blobs in an image
ž They can be seperated out and labelled

ž >>L = bwlabel(BW,n)
ž Returns a matrix L, of the same size as
BW, containing labels for the connected
objects in BW
ž n can have a value of either 4 or 8, where
4 specifies 4-connected objects and 8
specifies 8-connected objects; if the
argument is omitted, it defaults to 8

ž STATS = regionprops(L,properties)
ž Measures a set of properties for each
labeled region in the label matrix L
ž The set of elements of L equal to 1
corresponds to region 1; the set of
elements of L equal to 2 corresponds
to region 2; and so on

ž 'Area'– The actual number of pixels in the
region
ž 'Centroid'-- The center of mass of the region.
Note that the first element of Centroid is the
horizontal coordinate (or x-coordinate) of the
center of mass, and the second element is
the vertical coordinate (or y-coordinate)
ž 'Orientation' -- Scalar; the angle (in degrees)
between the x-axis and the major axis of the
ellipse that has the same second-moments
as the region. This property is supported
only for 2-D input label matrices

ž BW = imread('text.png');
ž L = bwlabel(BW);
ž stats = regionprops(L,'all');

ž Label into an RGB image for better
vizualization
ž RGB = label2rgb(L)

ž Binary area open remove small objects
ž BW2 = bwareaopen(BW,P)
ž Removes from a binary image all
connected components (objects) that
have fewer than P pixels, producing
another binary image, BW2.

ž OpenCV supports functions to calculate
moments upto order 3
CvMoments *moments = (CvMoments*)malloc(sizeof
(CvMoments));
cvMoments(img, moments, 1);

ž cvGetSpatialMoment(moments, xorder,
yorder)
ž cvGetCentralMoment(moments, xorder,
yorder)
ž Central = spatial/area

ž Example

ž For robotics purposes, moments are
fine till have one single object
ž If we have multiple objects in the same
binary image

ž You can think of contours as an
approximation of a binary image

ž You get polygonal approximation of
each connected area

ž The output you get for the previous
binary image is:
› Four “chains” of points
› Each chain can have any number of points
› In our case, each chain has four points

Contour plotting

ž Contour plot of an image im can be made
in MATLAB using the command:
ž im = imread(‘img.jpg');
ž imcontour(im,level)
ž Level=number of equally spaced contour
levels
ž if level is not given it will choose
automatically

ž OpenCV linked lists to store the
“chains”
ž We’ll see some code to find out the
squares in the thresholded image you
saw

CvSeq* contours;
CvSeq* result;
CvMemStorage *storage = cvCreateMemStorage(0);
• The chains are stored in contours
• result is a temporary variable
• storage is for temporary memory allocation

cvFindContours(img, storage, &contours, sizeof(CvContour),
CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0));
• img is a grayscale thresholded image
• storage is for temporary storage
• All chains found would be stored in the contours sequence
• The rest of the parameters are usually kept at these values
• Check the OpenCV documentation for details information about the
last four variables

while(contours)
{
result = cvApproxPoly(contours, sizeof(CvContour),
storage, CV_POLY_APPROX_DP,
cvContourPerimeter(contours)*0.02, 0);
• The previous command makes contours point to the first chain
• We’re approximating the contour right now
• After this command, result stores the approximate contour as a
polygon (many points)

if(result->total==4)
{
CvPoint *pt[4];
pt[i] = (CvPoint*)cvGetSeqElem(result, i);
}
• We’re looking for quadrilaterals, so we check if the number of
points in this particular polygon is 4
• Then, get extract each point using the command cvGetSeqElem
• Once you have the points, you can actually check the shape of the
object as well (by checking angles, lengths, etc)

// Do whatever you want with the 4 points
contours = contours->h_next;
}
• Do whatever you want to do with the four points
• Then, we move onto processing the next contour

ž MATLAB has an image acquisition
toolbox which helps capture images
ž Now-a-days most of the cameras are
available with USB interface
ž Once you install the driver for the
camera, the computer detects the
device whenever you connect it

ž In MATLAB, you can check if the
support is available for your camera
ž MATLAB has built-in adaptors for
accessing these devices
ž An adaptor is a software that MATLAB
uses to communicate with an image
acquisition device

ž COMMANDS:
ž >> imaqhwinfo
ž >> cam=imaqhwinfo;
ž >> cam.InstalledAdaptors

ž To get more information about the
device, type
ž >>dev_info =
imaqhwinfo('winvideo',1)
ž Instead of ‘winvideo’, if imaqhwinfo
shows another adaptor, then type that
adaptor name instead of ‘winvideo’.

ž You can preview the video captured by
the image by defining an object and
associate it with the device
ž >>vid=videoinput(‘winvideo’,1,‘RGB24
_320x240’)

ž Now to see the video insert the
following command:
ž >> preview(vid)

ž You should see a window pop-up, that
displays what your camera is capturing

ž The camera may support multiple video
formats. To see for yourself all the
supported formats, type
ž >>dev_info =
imaqhwinfo('winvideo',1);
ž >>celldisp(dev_info.SupportedForma
ts);
ž Check out for yourself the display of other
formats, by replacing `RGB24_320x240`
with other formats, in the definition of the
object vid

ž Now to capture an image from the
video, define the object vid as
described before and use getdata to
capture a frame from the video
ž >>start(vid); % This command
initiates capturing of frames and stores
the frames in memory
ž >>im=getdata(vid,1);
ž >>imview(im);

ž You can store the captured image as a
.jpg or .gif file using imwrite function
ž >>imwrite(im,'testimage.gif');
ž The image will be stored in
‘MATLAB71work’ folder

ž Static Processing
ž Step Processing
ž Real-Time Processing

ž Take a single picture of the arena and
process it
ž Find out critical regions and points
ž Apply some geometry and
mathematical calculations
ž Then blindly follow a specified path

ž Advantages
› Simplest to implement
› Can be fairly accurate with stepper motors
ž Disadvantages
› Bot goes blind because only one pic
determines the bot motion
› Accuracy is very low especially with DC
motors

ž Take images of the arena in discrete
intervals (say Eg. 10secs/image)
ž Process the images and find out critical
regions and points
ž Check bot orientation at these intervals
and try to correct
ž Partial feedback mechanism is
implemented

ž Advantages
› Quite simple to implement
› Can be very accurate with stepper motors
ž Disadvantages
› Bot goes blind for a particular period of
time
› Accuracy is compromised with DC motors
› Dynamic environmental changes cannot be
accounted for

ž Take images of the arena continuously at
a particular frame rate
ž Process the images and find out critical
regions and points
ž Check bot orientation at every frame and
correct
ž Complete feedback mechanism is
implemented

ž Advantages
› Very accurate for both DC and Stepper motors
› Gives dynamic feedback and accounts for
changing environment
› Can give bot orientation at each point of time
ž Disadvantages
› Requires more processing power
› Requires more memory for taking so many
images

ž Real-time image processing is the best
approach for any real application
ž Dynamic feedback systems give
excellent accuracy and precision and
hence is the best approach

ž Image processing is an important tool in
many applications
ž Problem is that one needs to acquire
images and pre-process them before
doing actual IP
ž Sometimes it may be required that offline
image processing is not possible i.e. one
needs to proceed with real-time IP or
even more, processing of the video itself

ž Every time you want to capture an
instantaneous image, you have to stop
the video, start it again and use the
getdata function
ž To avoid this repetitive actions, the
Image Acquisition toolbox provides an
option for triggering the video object
when required and capture an
instantaneous frame

ž Create an m-file with following
sequence of commands:

vid=videoinput('winvideo',1);
triggerconfig(vid,'manual');
set(vid,'FramesPerTrigger',1 );
set(vid,'TriggerRepeat', Inf);
start(vid);

for i=1:5
trigger(vid);
im= getdata(vid,1);
figure,imshow(im);
end
stop(vid);
delete(vid);
clear vid;

ž In the above code, object im gets
overwritten while execution of each of
the interations of the for loop
ž To be able to see all the five images,
replace im with im(:,:,:,i)

ž triggerconfig sets the object to manual
triggering, since its default triggering is
of type immediate
ž In immediate triggering, the video is
captured as soon as you start the object
‘vid’
ž The captured frames are stored in
memory. Getdata function can be used to
access these frames
ž But in manual triggering, you get the
image only when you ‘trigger’ the video

ž ‘FramesPerTrigger’ decides the number
of frames you want to capture each time
‘trigger’ is executed
ž TriggerRepeat has to be either equal to
the number of frames you want to
process in your program or it can be set
to Inf
ž If set to any positive integer, you will
have to ‘start’ the video capture again
after trigger is used for those many
number of times

ž Once you are done with acquiring of
frames and have stored the images,
you can stop the video capture and
clear the stored frames from the
memory buffer, using following
commands:
ž >>stop(vid);
ž >>delete(vid);
ž >>clear vid;

ž Getsnapshot function returns one
image frame and is independent of
FramesPerTrigger property
ž So if you want to process your images
in real-time, this is all you need:

vid=videoinput(‘winvideo’,1)
triggerconfig(vid,'manual');
set(vid,'FramesPerTrigger',1);
set(vid,'TriggerRepeat', Inf);
start(vid);
while(1)
{
trigger(vid);
im= getdata(vid,1);
% write your image processing algorithm here
%
% you may break this infinite while loop if a certain
condition is met
}

Using MATLAB

ž MATLAB provides support to access
serial port (also called as COM port)
and parallel port (also called as printer
port or LPT port) of a PC
ž MATLAB has an adaptor to access the
parallel port (similar to adaptor for
image acquisition)

ž To access the parallel port in MATLAB,
define an object
ž >> parport=
digitalio('parallel','LPT1');
ž You may obtain the port address using,
ž >> get(parport,'PortAddress')
ž >> daqhwinfo('parallel'); % To get
data acquisition hardware
information

ž You have to define the pins 2-9 as output
pins, by using addline function
ž >> addline(parport, 0:7, 'out')
ž Now put the data which you want to
output to the parallel port into a matrix;
e.g.
ž >> dataout = logical([1 0 1 0 1 0 1
1]);

ž Now to output these values, use the
putvalue function
ž >> putvalue(parport,dataout);
ž Alternatively, you can write the decimal
equivalent of the binary data and output
it
ž >> data = 23;
ž >> putvalue(parport,data);

ž You can connect the pins of the parallel
port to the driver IC for the left and
right motors of your robot, and control
the left, right, forward and backward
motion of the vehicle
ž You will need a H-bridge for driving the
motor in both clockwise and anti-
clockwise directions

ž Things are a little more involved
ž There is this library called inpout32
ž Makes your task really simple
ž Just follow the instructions that come
along, and you’ll be sending data to
your robot!

ž Or you could use the following code
ž The idea is:
› Create a virtual file that “represents” the
port itself (parallel, serial, etc)
› Keep this file open
› And keep writing to this file
› So, data is automatically send to the
desired port

ž Step 1: Create a global variable named
hPort
HANDLE hPort;

ž Step 2: Create function to create the
virtual file, and store its “handle” in hPort
(contd)
bool SerialOpen(LPCWSTR strPort)
{
// Open the serial port.
hPort = (HANDLE)CreateFile (strPort, // Pointer to the name of the port
GENERIC_READ | GENERIC_WRITE, // Access (read-write) mode
0, // Share mode
NULL, // Pointer to the security attribute
OPEN_EXISTING, // How to open the serial port
0, // Port attributes
(long)NULL); // Handle to port with attribute
// to copy
DCB PortDCB;
DWORD dwError;
// Initialize the DCBlength member.
PortDCB.DCBlength = sizeof (DCB);
// Get the default port setting information.
GetCommState (hPort, &PortDCB);

ž Step 2: Some more port configuration
(contd)
// Change the DCB structure settings.
PortDCB.BaudRate = 9600; // Current baud
PortDCB.fBinary = TRUE; // Binary mode; no EOF check
PortDCB.fParity = TRUE; // Enable parity checking
PortDCB.fOutxCtsFlow = FALSE; // No CTS output flow control
PortDCB.fOutxDsrFlow = FALSE; // No DSR output flow control
PortDCB.fDtrControl = DTR_CONTROL_ENABLE; // DTR flow control type
PortDCB.fDsrSensitivity = FALSE; // DSR sensitivity
PortDCB.fTXContinueOnXoff = TRUE; // XOFF continues Tx
PortDCB.fOutX = FALSE; // No XON/XOFF out flow control
PortDCB.fInX = FALSE; // No XON/XOFF in flow control
PortDCB.fErrorChar = FALSE; // Disable error replacement
PortDCB.fNull = FALSE; // Disable null stripping
PortDCB.fRtsControl = RTS_CONTROL_ENABLE; // RTS flow control
PortDCB.fAbortOnError = FALSE; // Do not abort reads/writes on error
PortDCB.ByteSize = 8; // Number of bits/byte, 4-8
PortDCB.Parity = NOPARITY; // 0-4=no,odd,even,mark,space
PortDCB.StopBits = ONESTOPBIT; // 0,1,2 = 1, 1.5, 2

ž Step 2: And finally…
// Configure the port according to the specifications of the DCB
// structure.
if (!SetCommState (hPort, &PortDCB))
{
// Could not configure the serial port.
dwError = GetLastError();
printf("Serial port creation error: %d", dwError);
MessageBox(NULL, L"Unable to configure the serial port", L"Error", MB_OK);
return false;
}
return true;
}

ž This function works equally well for
both serial ports and parallel ports

ž Step 2: I’ve assumed you’re creating a
function
ž This function returns a true when the
port is created successfully
ž If you’re not, replace the “return”
statements with “printf”s

ž Example usage of this function
› hPort = SerialOpen(L”COM8:”); // Serial Port
› hPort = SerialOpen(L”LPT1:”); // Parallel
ž This is actually how you can access ports
in DOS as well… using COM8: and LPT1:
instead of C:, D:, etc
ž The L before the quotes is just syntax for
C/C++

ž Step 3: Next, we’ll write functions to
write to the virtual file we’ve created
bool SerialWrite(byte theByte)
{
// The port wasn't opened
if(!hPort)
return false;
DWORD dwError, dwNumBytesWritten;
WriteFile (hPort, // Port handle
theByte, // Pointer to the data to write
1, // Number of bytes to write
&dwNumBytesWritten, // Pointer to the number of bytes written
NULL // Must be NULL
);
return true;
}

ž You pass the byte you want to write as
a parameter, and it gets written to the
port
ž Example: SerialWrite(12)

ž Step 4: And finally, a function to close
the port
ž Example: SerialClose()
bool SerialClose(void)
{
if(!hPort)
return false;
CloseHandle(hPort);
return true;
}

ž Again, I’ll emphasize that these
functions will work equally well for
both parallel ports and serial ports

ž A critical part of image segmentation
ž Used for detecting meaningful
discontinuities in intensity values
ž Done by finding first and second order
derivatives of the image
ž Also known as gradient of the image

ž There are different kinds of edge detectors
based on the criteria of derivatives
ž An edge detector can be sensitive to
horizontal or vertical lines or both
ž In detection we try to find out regions
where the derivative or gradient is greater
than a specified threshold

ž General Command in MATLAB
ž [g t] = edge(im,’method’,parameters);
ž g-gradient, t-threshold value

ž Different methods of edge detection

ž A threshold value can be given
manually as argument
ž g=edge(im,’method’,t);
ž t-threshold
ž Sobel& Canny are the more frequently
used edge detectors

ž Image Acquisition
ž Image Cropping
ž Image Filtering
ž Image Segmentation
ž Image Thresholding
ž Finding critical points (Centroids)
ž Finding bot centroidand orientation
ž Robot Feedback control
ž Robot Control

ž Static processing
ž Step processing
ž Real-time processing

ž Manual Cropping:
› I2 = imcrop(I);
ž Coordinate Cropping:
› I2 = imcrop(I,[x1 y1 x2 y2]);

ž Mean filter
ž Median filter
ž Histogram equalization

ž Use different kinds of edge detections
› Sobel
› Canny

ž Thresholding
› In RGB
› In HSV
ž Separating colors

ž Finding areas of blobs
ž Finding centroids of blobs

ž Using orientation

ž Control using parallel port

ž The idea of an orientation tag is to
precisely indicate the orientation of the
robot
ž It should happen with the least
number of operations

ž This orientation tag is bad
ž You can tell the position, but not the
direction the robot is facing

ž This orientation tag is excellent for a
single bot

ž But if you have a team of bots, this
won’t be the best choice
ž You need to have multiple colors for
each bot
ž So, you have more operations
ž Slowing down the program

ž A better choice for a team of bots
ž The asymmetry helps distinguish
between multiple bots, with just two
colors

ž Some more orientation tags

ž With C/C++ you get two methods to
capture images:
› Using OpenCV’s built in libraries
› Using some 3rd party capturing library

ž Use OpenCV functions
ž Use DirectX functions (on Windows)

Images, served in C/C++

ž OpenCV lets you access cameras
through the CvCapture structure
ž So we create a CvCapture structure

ž Tries to get access to cam #index
ž Useful when you have multiple
cameras attached to the same machine
(like in stereo vision)
ž Then we check if we were able to get
exclusive control of the camera. If not,
quit.

ž This window is for our convenience
ž Displaying what’s going on within the
program, what decisions are taken, etc

ž We’ll create a function to take snapshots
ž It will use the CvCapture structure to tap
into the camera’s stream
ž It will return a single frame as an
IplImage structure
ž Scroll down, and add the function

ž Now, we’ll display a live stream in the
window we had created
ž Because the getSnapshot() function
returns a single frame, we need to take
snaps regularly
ž So it goes into the do…while loop

ž If you try compiling the program right
now, you’ll get an error
ž getSnapshot() comes “after” the main
function
ž So the compiler doesn’t know if it
exists
ž Hence we need the so called
“prototype”

ž Once we’re done, we need to release

ž The cvCaptureFromCam works only for
some supported cameras
ž DirectX is a much bigger library and
supports almost all cameras that exist

ž First, download the Microsoft Platform
SDK for Windows Server 2003 R2
› http://www.microsoft.com/downloads/deta
ils.aspx?familyid=E15438AC-60BE-41BD-
AA14-7F1E0F19CA0D&displaylang=en

ž Second, download the DirectX SDK
ils.aspx?FamilyID=77960733-06e9-47ba-
914a-844575031b81&DisplayLang=en

ž Third, download the DirectShow SDK
ils.aspx?FamilyID=8af0afa9-1383-44b4-
bc8b-7d6315212323&DisplayLang=en

ž Fourth, download the VideoInput
library
› http://muonics.net/school/spring05/videoI
nput/

ž These are external libraries. So we need
to tell Visual Studio where to find the
required external files
ž So we follow steps similar to the OpenCV
ones.
ž The VideoInput package comes with lots
of sample code, so you shouldn’t have
much problem

ž You can check some sample code at
this website as well
› http://opencv.willowgarage.com/wiki/Direc
tShow

ž For real time, you need to process
captured frames as quickly as possible
ž So we use a loop of some kind (usually
a do…while)
ž Within the loop, you do the following
tasks:

ž Task 1: Capture an image
ž Task 2: Pre–processing it
ž Task 3: Process the image
ž Task 4: Take a decision
ž Task 5: Move the bot (with feedback)
ž Task 6: Go to Task 1

ž Quite obvious
ž You need an image
ž Only then can you process something

ž Once you have the image, you need to
enhance it with pre-processing
ž Increase contrast, reduce noise,
smoothen it out, etc

ž With the pre-processed image, you
figure out the location of each object in
the arena
ž Use moments, contours, or anything
else
ž Thresholding, morphology, etc are
helpful here

ž With the location of each object, you
can decide what to do next
ž Bot: Should I go to the red ball
because it’s the closest? Or should I go
to the green ball because it has the
maximum number of points

ž Once decided where to go, you make
the robot move
ž You must include feedback in this step
itself (maybe another do…while loop)

ž You have two options:
› Check if further movement is necessary
(have all the balls been potted?) If
required, only then go to Task 1
› Blindly go to Task 1 (the decision module
will tell the bot when to stop)
ž Both options are good enough

ž Task 1: Capturing the image
ž Task 2: Pre-processing the image
ž Task 3: Processing the image

ž Task 4: Taking a decision
ž Task 6: Go to Task 1

ž Task 5: Move the bot (with feedback)
ž We won’t go into code. Just high level
logic on how to go about it

ž For the simplest feedback mechanism,
you use pixels
ž Everything is measure in terms of
pixels: distances, coordinates, etc
ž Angles are usually represented in
radians (cos, sin, etc work well with
radians)

ž First, you need to decide your
coordinate system
ž And you need to stick with it
throughout your code
ž Here’s a coordinate system I used
several times

ž This is a top-down view of the arena,
just like the camera sees it

ž So all your calculations MUST use this
particular coordinate system
ž In particular, you must make sure all
your angle calculations are consistent
ž And that they cycle through 359-1
degrees perfectly

ž Possible function you might want to
create:
› MoveBotToPosition(x, y)
› TurnBotToAngle(angle)

ž TurnBotToAngle(angle)
› Turns the bot to angle degrees of the
coordinate system
› This would be the very basic feedback
function
› Within this function, you have a loop
› This loop keeps running as long as the bot
isn’t oriented at angle degrees
› You’ll take snapshots within this function
as well

ž MoveBotToPosition(x,y)
› Moves the bot until it reaches the
coordinates (x,y) in the image
› If required, you can also put a call to
TurnBotToAngle (to orient the bot to move)
› Again, there’s a loop which keeps running
until the bot reaches the desired position
› And you’ll need to take multiple snapshots
to check where the bot actually is

ž You obviously need to set a range
ž If you say TurnBotToAngle(30), it’s
very unlikely that the bot will orient to
exactly 30 degrees
ž A range, say 28-32 should be good
enough

ž Reminder: All x, y and angle we’ve
talked about are in the image, in terms
of pixels
ž We have no idea how they relate to
physical distances and angles
ž But we’re sure that they are
proportional to physical distances and
angles

ž Though, you CAN calibrate your
camera and actually figure out physical
distances
ž For example, 5pixels = 3cm
ž This is very much possible, but of no
use for our purposes!

ž Use classes and structures as much as
possible. And you don’t need to know
OOPs to use them.
ž They really simplify your work, and
even make the code more readable

ž What we’ve described requires that the
bot stops and then checks if the angle
is correct or not, etc
ž Try working on something which
checks the bot’s angle without
stopping the bot

ž Working in the YUV space (this is what
cameras use… so thresholding in YUV
itself will eliminate processing time
consumed by YUV to RGB conversion)

ž Kalman filters: If a bot is static, still it’s
position might be calculated as
different for different frames.
ž If you use Kalman filters, you can
“smooth out” the position data and get
precise positioning and angle data

ž Can you eliminate thresholding
altogether?
ž Yes you can!!!
ž How, you ask? Figure it out! Its
EXTREMELY simple!

ž You now know enough to participate in
image processing based competitions
ž All this knowledge can even serve as a
start point for further studies in image
processing
ž Enjoy!

ž Utkarsh Sinha
ž Ajusal Sugathan
ž Oh, btw, visit http://liquidmetal.in/ for
more!

The RoboCV Workshop

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to The RoboCV Workshop (20)

Recently uploaded (20)

The RoboCV Workshop