Computer vision and robotics

Presentation Outline
• Computer Vision
– What is Computer Vision?
– Applications of Computer Vision
– Face Recognition
– Challenges in Computer Vision
– The Future
• “Image Analysis for Ethiopian Coffee Classification”
• Robotics
– What is Robotics?
– Definition of a Robot
– History of Robot
– Types of Robots
– The Purpose of Robots
– Use of Robots
– Example of Robots
– The way forward [The Future]
• “Robotics – Chances and Challenges of a Key Science”

Introduction
• Image is understood intuitively as the visual response
on the retina or light sensitive chip in a camera, TV
camera or another device.
• The three related disciplines studying images are:
– Digital Image Processing – 2D static world, no image
interpretation involved (rather independent of an
application domain), signal processing techniques.
– Image Analysis – 2D world, image interpretation involved,
i.e. image interpretation constitutes the crucial step.
– Computer Vision – the most general problem
formulations, 3D world, interpretations, potentially
dynamic (i.e., image sequence needed), ill posed tasks
very ambitious.

What is Computer Vision?
• Different scholars define the concept ‘Computer Vision’ in
different ways.
– Trucco and Verri: computing properties of the 3D (three
dimensional) world from one or more digital images
– Sockman and Shapiro: To make useful decisions about real
physical objects and scenes based on sensed images
– Ballard and Brown: The construction of explicit,
meaningful description of physical objects from images
– Forsyth and Ponce: Extracting descriptions of the
world from pictures or sequences of pictures

The goal of computer vision research:
• to provide computers with human-like
perception capabilities so that:
– they can sense the environment
– understand the sensed data
– take appropriate actions, and
– learn from this experience in order to enhance
future performance.
• Computer Vision is making the machine see as
we do! It is a challenging task.

… Introduction
• Computer vision is probably the most exciting
branch of image processing, and the number of
applications in robotics, automation technology
and quality control is constantly increasing.
• One of the most common uses of machine
(computer) vision in robotic agents is to identify
objects in the agent’s path.
– One of the methods -> parts decomposition method

The major reasons for studying
Computer Vision
• Images and movies are everywhere
• Fast-growing collection of useful applications
– building representations of the 3D world from pictures
– automated surveillance (who’s doing what)
– movie post-processing
– face recognition
• Various deep and attractive scientific mysteries
– How does object recognition work?
– Beautiful marriage of math, biology, physics,
engineering
• Greater understanding of human vision

The steps in computer vision
• Image acquisition
• Image manipulation
• Image understanding
• Decision making

Two areas of Computer Vision
1. Image Enhancement:
refers to the accentuation or sharpening of image
features such as edges, boundaries or contrast to
make an image more useful for display and analysis.

… Two areas of Computer Vision
2. Image Analysis:
The goal is to extract important features from image
data, from which a description, interpretation, or
understanding of the scene can be provided.

Applications of Computer Vision
• Automotive:
– Driver assistance systems.
– Reading automobile license plates, and traffic management.
– Lane departure warning systems.
– Head tracking systems for drowsiness detection.
• Photography:
– In camera face detection, red eye removal, and other functions.
– Automatic panorama stitching.
• Games:
– Tracking human gestures for playing games or interacting with
computers.
– Tracking the hand and body motions of players
– Image-based rendering, vision for graphics.

… Applications of Computer Vision
• Movie and video (a very big industry):
– Multiple cameras to precisely track tennis and cricket balls.
– Human expression recognition.
– Software for 3-D visualization for sports broadcasting and analysis.
– Tracking for character animation.
– Augmented reality.
– Tracking objects in video or film and solving for 3-D motion to allow for
precise augmentation with 3-D computer graphics.
– Tracking consistent regions in video and insert virtual advertising.
– Motion capture, camera tracking, panorama stitching, and building 3D
models for movies.
• Industrial automation (a very big industry):
– Vision-guided robotics in the automotive industry.
– Electronics inspection systems for component assembly.

… Applications of Computer Vision
• General purpose:
– Image retrieval based on content.
– Inspection and localization tasks, people counting, biomedical, & security etc.
– Object recognition and navigation for mobile robotics, grocery retail, and
recognition from cell phone cameras.
– Laser-based 3D vision systems for use on the space shuttles and other
applications.
• Medical and biomedical (maturing):
– Vision to detect and track the pose of markers for surgical applications,
needle insertion, and seed planting.
– Tele operations
– Quantitative analysis of medical imaging, including diagnosis such as cancer.
• Security and biometrics (thriving/ flourishing):
– Biometric face, fingerprint, and iris recognition.
– Intelligent video surveillance.
– Behavior detection.

Face Recognition
• One very popular research area of computer
vision at present is the study of automatic
face recognition.
• This problem is an excellent example of the
kinds of problems that Artificial Intelligence
techniques are usually applied to.

Difficulties with Automatic Face Recognition
– the conditions in which a face can be seen
• lighting, distance from camera to face, and angle
– human faces are so flexible and so capable of being
altered.
– Facial expressions
– people also are able to grow beards; cut or grow their
hair; wear glasses, sun-glasses, hats, and earrings; and
grow older
↔ identifying invariant properties of a given face is an
important first step in automating the process of
face recognition.

Approaches to Face Recognition
I. To identify particular facial features
– such as eyes, nose, mouth, eyebrows, and so on,
and to store information about the relative
positions of those features.
– works in some circumstances, but is not
particularly robust.
– PROBLEM: it assumes that the best way to tell the
difference between two faces is to note the
locations of the features such as eyes, mouth, and
so on. This might not be the case.

… Approaches to Face Recognition
II. Eigenfaces
 are based on the idea of principle component
analysis.
 Principle component analysis:
 determine the features of the data that vary most
from one item to another.
E.g. the position of the tip of the nose relative to the end
of the chin is the feature that varies the most
 a very robust way of recognizing faces
(Accuracy > 90%)

Challenges in Computer Vision
• Loss of information in converting 3D to 2D due to perspective
transformation (Mathematical abstraction = pinhole model).
• Inherent presence of noise as each real world measurement
is corrupted by noise.
• A lot of data [Storage Space Requirement]
– Sheet A4, 300 dpi, 8 bit per pixel = 8.5 Mbytes.
– Non-interlaced video 512 × 768, RGB (24 bit) = 225 Mbits/second.
• Interpretation needed
• Measured brightness is given by complicated image
formation physics.

The Future
• Research and application by the image
understanding (IU) community suggests that
the most fruitful approaches to IU involve:
– analysis and learning of the type of information
being sought, the domain in which it will be used,
and systematic testing to identify optimal
methods.

“Image Analysis for Ethiopian
Coffee Classification”
By Habtamu Minassie (2008)

… Image Analysis for Ethiopian Coffee Classification
• Motivation
– In Ethiopia, technologies of image analysis or
computer vision have not been explored in a
significant manner in the development of
automation in agricultural and food industries.
– Particularly, Ethiopian coffee quality inspection is
based on traditional ways of classification and
grading system.

• Problem Statement
– There is a need for automated inspection, as well
as identification systems so that the abuses during
distribution and marketing can be minimized.
– human perception could easily be biased.
– Therefore, this thesis work initiated a model for
Ethiopian coffee variety classification which is
consistent, efficient and cost effective by exploring
the technology of image analysis.

• Objective
– to design an appropriate classification model of Ethiopian
coffee varieties with respect to their growing region or
area of plantation by using image analysis techniques.
Schematic Representation of Coffee Classification Procedure

• The classification process
involves:
1) Image acquisition of a
coffee bean from each
growing region
• images have been taken from six
major coffee export producing
regions of Ethiopia.
• They are Bale, Harar, Jimma,
Limu, Sidamo and Welega.

2) An image processing techniques
 is applied on the acquired image to enhance the
quality of image so as to remove noises.
 It includes all tasks in image pre-processing,
image segmentation and removal of noises.
– ImageJ was used for image processing tasks of coffee
bean images to enhance the quality of image and to
change images to binary for feature extraction
purposes.
– Then, images were segmented by using histogram
based thresholding technique.

3) Appropriate features are extracted
• For image analysis of Ethiopian coffee, two
classification parameters are identified.
i. Morphology is the geometric property of images.
the size and shape characteristics of coffee beans. It can
be obtained from the analysis of binarized images.
ii. The color features are extracted by computing the mean
values of RGBs and HSIs of coffee bean images.
In summary, some of the sixteen features (ten morphological
and six color features) were used for the classification of
Ethiopian coffee beans by the growing region.

4) The classification model
 To test the classification accuracy of each
selected feature set, Naïve Bayes and Neural
Network classifiers were compared.
The experiment was conducted under three scenarios
of the features data set such as morphology, color and
both morphology and color features.
5) Finally, suitable pattern classifiers are selected to
classify Ethiopian coffee beans to the predefined
classes of the growing region.

Performance Registered
• The best classification accuracy was obtained
using neural networks when morphology and
color features were used together.
• The six regions, Bale, Harar, Jimma, Limu, Sidamo
and Welega, were identified with the
classification accuracy of 80.7%, 72.6%, 56.8%,
96.8%, 95.4% and 61.9% respectively by using
neural network classification with morphological
and color features parameters.
• In this case, the overall performance was 77.4%.

Recommendations
• In Ethiopia, no research has been conducted in this
direction to support the agriculture sector. Hence, the
research may pave the way and initiate researchers to
work more in the area.
Computer vision for coffee defect identification and
counting
Coffee grading by using computer vision system
Identification of coffee varieties from mixed components
of coffee beans
Coffee classification by using image analysis of roasted
coffee

Robotics
• Robotics is the branch of technology that
deals with the design, construction, operation,
structural disposition, manufacture and
application of robots.
• Robotics is related to the sciences of
electronics, engineering, mechanics, and
software.
• The term robotics was first used by Isaac
Asimov in 1950.

Definition of a Robot
• The Formal definition given by the Robot Institute of
America is:
"A re-programmable, multifunctional
manipulator designed to move material, parts,
tools, or specialized devices through various
programmed motions for the performance of a
variety of tasks".
• “Robots” is physical agents that perform tasks by
manipulating the physical world.
– They are equipped with sensors to perceive their
environment and effectors to assert physical forces on it.

History of Robot
• The word robot was popularized by Czech
playwright Karel Capek in his 1921 play R.U.R.
(Rossum's Universal Robots).
• The first industrial robot was installed in 1961.
These are the robots one knows from
industrial facilities like car construction plants.

Types of Robots
• Industrial Robots
• Materials handling
• Welding / Repairing
• Inspection
• Improving productivity
• Laboratory applications
• Mobile Robots
– Robots that move around on legs, tracks or
wheels.

… Types of Robots
• Educational Robots
• Domestic Robots:
(i) Those designed to perform household tasks
(ii) Modern toys which are programmed to do things like
talking, walking and dancing.
• Hardware Robots
– Non-adaptive robots have no way of sensing the
environment, so do the job regardless of any
environmental factors
– Adaptive Robots: get feedback from a sensor to alter the
operation of the device.

The Purpose of Robots
• Repetitive tasks that robots can do 24/7
• Robots never get sick or need time off.
• Robots can do tasks considered too dangerous for
humans.
• May be able to perform tasks that are impossible
for humans.
• Robots can operate equipment to much higher
precision than humans.
• May be cheaper over the long-term
• To assist handicapped people

Use of Robots
• Exploration
The hardest thing any robot has to do is to be able to taught
how to walk.
– Space Missions
– Robots in the Antarctic
– Exploring Volcanoes
– Underwater Exploration
• Medical Science
– Surgical assistant
• Assembly
– Factories

Example of Robots
Shakey (1966 – 1972)
• Shakey was the first mobile robot with
the ability to reason and react to its
environment.
• Developed at SRI’s (Stanford Research
Institute) pioneering Artificial Intelligence
Center, Shakey has had a substantial
influence on present-day AI and robotics.
• Using a TV camera, a triangulating range
finder, and bump sensors, Shakey was
connected to DEC PDP-10 and PDP-15
computers via radio and video links.

…Example of Robots
Centibots (2002 – 2004)
• The Centibots are mobile, coordinated
robots that can autonomously and
effectively explore, map, and survey the
interior of unknown building structures.
• The Centibots marked a milestone in
robotics, representing the largest
collection (more than 100) to date of
coordinated autonomous mobile
robots.
• These autonomous team robots were
designed to augment the situational
awareness of human teams – such as
crisis response teams – in situations
that could pose a threat to people.

Trauma Pod and Medical Automation
Robots (2005 – present)
• SRI is the lead integrator on a collaborative DARPA program to develop
a futuristic battlefield-based, unmanned medical treatment system
dubbed the “Trauma Pod”.

• SRI’s M7 surgical robot conducted the first-ever
acceleration-compensated medical procedure in zero-
gravity flight for NASA.
• The M7 was also the first surgical robot to be successfully
deployed to an undersea habitat simulating the rigors of
outer space in NASA’s Extreme Environment Mission
Operation (NEEMO), demonstrating remote surgery over
1,200 miles of public Internet.
• One year later, the M7 demonstrated the first autonomous
ultrasound-guided medical procedure in the same undersea
laboratory.
M7

• SRI’s wall-climbing robots scale vertical
surfaces by virtue of electroadhesion (an
electrically controllable adhesion technology),
which involves inducing electrostatic charges
on a wall substrate using a power supply
connected to compliant pads situated on the
moving robot.
• They have military or commercial
applications in the inspection of bridges,
containers, pipes and storage tanks, buildings,
structural walls, ducts, aircraft and ship hulls,
and transmission towers.
• Wall-climbing robots could also be operated
for cleaning windows and for painting
buildings, bridges, or aircraft.
Wall-Climbing Robot (2007 – present)

Software Solutions for Robot Platforms -
“KARTO” (2007 – present)
• SRI’s KARTO™ robot mapping technology provides advanced
mapping and localization software.
• It enables developers of mobile robot solutions to integrate
navigation and mapping intelligence into their designs using
various robotic platforms and development environments,
including Microsoft Robotics Studio.

Mini-Andros
• The Mini-Andros is used by bomb squads
across the country to locate and dispose of
bombs.

The way forward [The Future]
• Although most robots in use today are designed for specific tasks, the goal
is to make universal robots, robots flexible enough to do just about
anything a human can do.
• Visual object recognition:
Our robots today are not very aware of their surroundings, as we do not
have general- purpose vision algorithms that can recognize particular
objects never seen before as an instance of a known class.
• Manipulation:
We need to develop widely deployable robot hands so that hundreds of
researchers can experiment with manipulation.
• New sensors:
Direct investment in new sensor modalities for robots will lead to new
algorithms that can exploit them and make robots more aware of their
surroundings, and hence able to act more intelligently.

… The way forward [The Future]
• Materials science: is producing radically new materials with
sometimes hard‐to‐believe properties.
• Distributed and networked robots: Technology allows us to
decompose tasks in ways that humans are incapable.
• Awareness of people: Most future applications of robots will
require that they work in close proximity to humans. To do so
safely, we need both perceptual awareness of people, and
actuators and robots that are intrinsically safe for humans to
physically contact.
• Social interaction: If ordinary people are to work with robots
they must be able to interact with them in cognitively easy
ways.

The practical application domains where
robotic technology is most likely to be used:
• Transport (public and private)
• Exploration (océans, space, deserts, etc.)
• Mining (dangerous environments)
• Civil Defense (search and rescue, fire fighting
etc.)
• Security/Surveillance (patrol, observation and
intervention)
• Domestic Services (cleaning etc.)
• Entertainment (robotic toys etc.)
• Assistive Technologies (support for the fragile)
• War Machines
• Scientific Instrumentation (e.g. synchrotron
sample preparation, chemical screening etc.)

Robotics – Chances and
Challenges of a Key Science
By G. Schweitzer

Objective
• The paper gives examples of the actual state
of the art by referring to nano-manipulation,
a human leg prosthesis, and by looking at
developments in the medical area, and into
embedded robotics.
• The paper presented some aspects and results
and comments on robot intelligence, on expected
benefits of future robot technology, as well as on
socio-economic, legal and ethical constraints.

Trends and Potential Benefits of
Robotics
1. Technology
• It can be stated that any technical progress in
robotics will quickly spread over to products of
everyday life and may eventually initiate further
progress.
• For example, Automotive technology for
modern cars in making advanced use of
sensors for controlling their dynamics and
assisting in safe driving are following ideas from
robotics.

… Trends and Potential Benefits of Robotics
2. Biology and neuro-science
• Robotics has a very stimulating co-operation
of mutual interest with biological information
processing and neuro-science.
• For example, investigations on
– the walking of stick insects,
– the navigation of desert ants,
– the swimming motion of fish,
– the flight techniques of bees
are being related to mobile robots.

3. Man-machine relations
a) Communication and emotional
behavior
• The activity and mobility that can
be exerted by a robot will allow a
wider range of communication
modalities.
• The robot can turn its attention
actively to points of interest, it can
explore strange situations, and it
can actually “bring” information or
objects.
Aibo
Sony’s robo-dog with emotional behaviors

b) Allocation of work and authority
• It appears natural to design machines that can co-
operate in an "intelligent" way with their human
users, thus extending the range of the human and
making best use of the capabilities of the machine.
Still utopic scenario
of a robot helper
assisting a human
expert in sheet
bending

c) A change of paradigm in robotics objectives
• It appears to be most essential to design
the relation between robot and human
as co-operative and not as competitive.
“Instead of building machines that can do the
work of humans, we should build machines that
can do the work which humans cannot do, or do
not want to do”

4) Legal and ethical aspects and challenges
4.1. Legal challenges
It is easily conceivable that the exceptional situations
during such a robot task cannot be completely predicted,
that malfunctions cannot be excluded, and that the risk
of humans being injured by such a robot is immanent.
Who would be responsible?
4.2. Philosophical and ethical issues
The ideas about what machines could or should do to make
them “behave” in a complementary or even similar way to
human beings touch upon a broad range of human values.

5) Applications and examples
Intelligent robots will offer novel chances in
various ways and for different areas.

Intelligent control of leg prosthesis as an
example for medical applications
• The above-knee prosthesis
shows the direct interaction
between man and machine.
• The prosthesis can be seen as
a robot that is, at least
partially, replacing human
walking functionality.

Computer vision and robotics

More Related Content

What's hot (20)

Similar to Computer vision and robotics (20)

More from Biniam Asnake (6)

Recently uploaded (20)

Computer vision and robotics