Embedded_System_and_Smart_Phone_based_Object_Recognition_Technique_for_Visually_Impaired.pdf

2022 IEEE World Conference on Applied Intelligence and Computing
978-1-6654-7988-2/22/$31.00 ©2022 IEEE 293
DOI: 10.1109/aic.2022.49
Embedded System and Smart Phone based Object
Recognition Technique for Visually Impaired
Kapil Rajput, Sanjeev Kumar Shah, Ajay Prasad Nautiyal Dept.
of ECE
Uttaranchal University, Dehradun, India
kapilrajput1984@gmail.com, sanjeevkshah19@gmail.com,
ajaynautiyal16@gmail.com
Mohd. Noushad
Dept. of CSE
Shri Ram Group of Colleges, Muzaffarnagar, India
naushad47@gmail.com
Abstract- The goal of this work is to help the blind person to
recognize an object clearly. An integrated sensor and Artificial
Intelligence based approach has been used in this work. The
system consists of two modules, one module contains all
sensors interfaced with microcontroller unit and another
module contains android applications. The proposed system
uses Ultrasonic Sensor, Water Sensor, Fire Sensor and
Arduino UNO ATmega328 microcontroller in integrated
module. These sensors are used here to detect the presence of
object, water and fire respectively, near by the visually
disabled person. Here, the Talkback and select to speak
feature are used to connect an internet in smart phone. It will
also send the audio, video call, multimedia message and GPS
location on the android mobile using Google assistant
application. The object detection application in android
mobile will provide the exact object recognition facility and
output of ODA will be heard in wireless headphone.
Keywords: Ultrasonic Sensor, Water Sensor, IR Flame Sensor,
Arduino UNO ATmega328 Microcontroller, Multimedia
Message, Android Mobile, Android Applications, Artificial
Intelligence, ODA app, Wireless Headphone.
I. INTRODUCTION
Globally 2.2 billion people are pompous from the near and
distance vision impairment in the current situation. This
kind of system will assist the blind human to navigate from
one place to another safely. Previously, some of the works
have been reported on this approach are as follows. Three
ultrasonic sensors are used here to detect the obstruction, a
buzzer is used to make a person aware [1]. This kind of
stick used the GPS and GSM module along with the
Infrared and ultrasonic sensor, to find an object and
obstacle. This stick has three ultrasonic sensors at different
position to detect the object. In case if the blind person has
lost his stick, RF module is used here to get the stick once
again [2]. In another reported work the stick has E-SOS
button and three ultrasonic sensors at three sides to detect
the object [3]. This kind of system also has the ultrasonic
sensor to detect the object up to 5 meters. The BTS system
is used to deliver telecommunication services to people
who are blind. The GPS system provides the exact location
of the lost person [4].
This paper proposes the development of a baton to provide
detection of obstruction like as water and fire on the way
of visually disabled person through ultrasonic sensor HC-
SR04, water sensor and IR flame sensor in the integrated
module [5]. An ODA application is used in the system to
recognize the object exactly with its name in audio from.
Another part of this system is an android application
module to help the visually disabled person.
The goal of this complete system is to provide the correct
details regarding object in front of blind persons. This
approach will also help the blind person to provide the
calling functionality as Well as GPS coordinates sending
facility using Google assistant application. If the blind
person feels that he has lost his path, then he can send the
GPS location coordinates to his relative using Google
assistant application. This technology is enough to provide
the proper guidance to visually disabled person. The
novelty of our system is that we have interfaced all the
sensors in integrated module with one ATmega328 micro-
controller and use of ODA application in android mobile
[6]. An ODA application is used in our system to
recognize the object exactly with its name up to 50 meters.
Another feature is a Google assistant in this system that
makes our system magnificent because in critical situation
the visually disabled person can accomplish many tasks by
this application.
This paper proposes the development of a baton to provide
detection of obstruction like as object, water and fire on
the way of visually disabled person through ultrasonic
sensor HC-SR04, water sensor and IR flame sensor on the
integrated module.
II.METHODOLOGY
An ultrasonic sensor is used in this baton to detect obstacle
in front of a visually disabled human, the water
identification sensor involves here to gauge the water level
and IR flame detection sensor incorporates here to identify
the fire nearby the blind person. The Arduino UNO
ATmega328 microcontroller has been programmed to
calculate the distance from an object, to measure water
level and to recognize the fire near to the blind person [7].
The Android mobile with an Object Detection Application
2022
IEEE
World
Conference
on
Applied
Intelligence
and
Computing
(AIC)
|
978-1-6654-7988-2/22/$31.00
©2022
IEEE
|
DOI:
10.1109/AIC55036.2022.9848952
Authorized licensed use limited to: VIT-Amaravathi campus. Downloaded on May 29,2023 at 17:20:45 UTC from IEEE Xplore. Restrictions apply.

294
has been deployed at the top of the stick to recognize the
object in the form of plain text and as well as voice. The
mobile phone has been connected to the wireless
headphone by which the visually impaired person can
easily hear the name of the object in front of him.Fig.1. In
two parts A and B demonstrates the Overall pattern of the
proposed structure.
(A) Integrated Sensor Module
(B) Google Assistant and ODA Module
Fig .1 Module A and B display the overall pattern of the proposed
system
The mobile phone also provides the audio and video calling
and GPS functionality to the blind person by using the
Google assistant application [8].
III. EXPERIMENTAL SETUP
Testing of the circuit and simulation have been performed
on the Lab center Proteus professional 8.0. The Proteus 8.0
is used to test the Arduino UNO ATmega328
microcontroller with different kind of sensors like ultrasonic
sensor, water sensor, fire sensor in integrated module [9].
Fig.2 demonstrates the implementation of the circuit
diagram of the proposed structure on Proteus simulator.
Fig.2. Circuit Simulation in Proteus Professional
After the simulation has been done on Proteus professional
8.0, the original structure of the system has been developed
using hardware components. It is observed that here
ultrasonic sensor has been used here to detect the obstacle
in front direction [10]. A water sensor works here to
identify water level and fire sensor has been used to
provide the detection of fire near by the blind person [11].
It is found that the simulation results got reproduced in
actual implementation. The development and testing of an
android application have been done on android studio and
emulator and it is shown in Fig 3.
Fig.3. Demonstration of ODA on android studio and emulator
The ODA application identifies the object with the exact
name in text form as well as in audio form and the blind
person can hear this output into the Bluetooth connectivity
headphone. The stick is having the android mobile that
provides the calling and GPS location sending facility
through the use of Google assistant.
IV. IMPLEMENTATION OF PROPOSED
SYSTEM
The microcontroller is the core component of this smart
object recognition system that is used to control all the
sensors [12]. The power supply for this microcontroller is
provided by a 9-volt battery. The ultrasonic sensor has been
used here to recognize the object. An ultrasonic sensor's
trigger pin receives a 10 microsecond TTL pulse. [13]. After
this the sensor module transmits a 40 Hz ultrasonic sound
wave and if this wave strikes an object, then these signals
are returned back as an echo signal by which we are
computing the distance between target object and sensor
based on the period between sending a signal and receiving
an echo signal [14]. The total distance between an object
and sensor is calculated by using the travel-time.
𝑇𝑜𝑡𝑎𝑙 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒(𝐷) =
1
2
(𝑣 × 𝑡)
Where𝑣 shows velocity of ultrasonic sound waves and
𝑡 shows thetime interval between incident and reflected
wave.
A water sensor is available here to measure water level.

295
Water detector is a compact electronic-circuit that is used
to recognize the existence of aqua [15]. At a point where it
actually touches the water. The water sensor circuit
becomes short circuited and the whole circuit has been
shut down and a sign of the presence of water is produced
through the buzzer. The series of exposed parallel
conductors together works as a variable resistor whose
resistance shifts according to the water level [16]. The
water level depends on the resistance of the water that can
be understood by the following relation.
𝑊𝑎𝑡𝑒𝑟 𝑙𝑒𝑣𝑒𝑙 ∝
1
𝑅𝑒𝑠𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑜𝑓 𝑤𝑎𝑡𝑒𝑟
The Water level sensor works to detect the water level as in
a capacity tank. These tests send data back to the MCU
board to trigger an alarm. The blind person can identify the
presence of water by hearing the sound of buzzer.IR Flame
Sensor is used here to determine the existence of fire in
nearby place of a visually disabled person by which a blind
person can be aware from the fire and be ready to handle
this kind of situation in advance [17]. The buzzer alarm
system has been used here to provide the indication of fire,
to the blind person. Fig.4 Flowchart shows the sensor
module of the proposed system.
Fig.4. Flowchart: The Working of All Three Sensor Used in Such Stick.
The ODA application has been developed in android
studio using kotlin and executed on an emulator [18]. The
Google ML kit has been used in this application to
recognize the object correctly. The ODA application has
been used here to identify the object with its name into the
text and voice form [19]. This voice will be heard to the
blind person into wireless headphone. Fig.5. shows a
flowchart for the working of an ODA and Google assistant
application.
Fig.5. Flowchart to demonstrate working of ODA and Google Assistant

296
The visually impaired person can use the Google assistant
application when an internet connection will on [20]. The
visually disabled person will connect his internet connection
with the help of Talkback and Select to speak features of
android mobile. Fig. 6 shows the live demonstration of
talkback and select to speak to connect the internet. The
Talkback feature of mobile will speak all the things that he
wants to do in his mobile. Single click is used in Talkback
to select any icon and double click is used to open that icon.
Practically, the use of Talkback and Select to speak feature
to connect the internet are working well. During Talkback,
the double finger will be used to slide the menu in mobile.
There are two options for the blind person to connect the
internet with his mobile. One is from the settings of mobile
and another way by using the shortcut menus. When the
visually disabled person opens his shortcut menu by sliding
from the bottom of mobile with double fingers, then he can
select the internet data icon from this menu.
Fig.6. Shows the use of Talkback and Select to speak to connect internet
When he will select it, the Talkback application will speak
the icon details such as "select data”. Select to speak draw a
small icon in the lower right corner of the Smartphone.
When the visually disabled person touches the lower right
corner of Smartphone then he has to press that small icon
two times. As a result, the internet will be on in his mobile.
Now the blind person can easily use both the android
applications, Google assistant and ODA. Now the visually
disabled person can make an audio call, video call as well as
he can also share a message and GPS location. The ODA
application will be opened by visually disabled people with
the help of Google Assistant. The blind person will use his
voice to do his task. Now, the visually disabled person can
speak the ODA application after “OK Google” then this
ODA application will open. Similarly, he can complete the
rest of the task, such as calling and GPS location sharing.
These applications provide lots of functionality to the
visually disabled person to navigate easily.
V. RESULT AND DISCUSSION
The proposed design has been created and implemented
for the practical applications. The programming code has
been loaded to the Arduino UNO ATmega328
microcontroller. To use a microcontroller with all sensors,
we need a power supply. The power to the system will be
provided through 9-volt battery. This smart stick is used to
fulfill the 95% task by the blind person correctly. We have
used an ultrasonic sensor to detect a barrier in front of a
person. Table 1 Shows a comparison between the previous
reported works and the proposed work.
TABLE.1. DEMONSTRATE THE COMPARISION
BETWEEN PREVIOUS FINDING AND OUR SYSTEM
S.
No
Previous Paper’s Findings Present Paper Findings
1. “Ultrasonic Sensor Based
Smart Blind Stick”
Sensing scheme:
Three ultrasonic sensors with
PIC 16F877A and buzzer have
been used. It can detect objects
from 5cm to 35 cm. It provides
no calling and GPS facility.
“The proposed work”
Sensing scheme:
Three different types of sensors
as Ultrasonic, Water, IR Flame
Sensor and buzzer have been
used with ATmega328 MCU. It
can detect distance, water and
flame in an area of 0.5cm to 200
cm. It provides Calling, ODA
and GPS facility.
2 “Embedded Assistive Stick
for Visually Impaired
Persons” Sensing scheme:
3 ultrasonic, 1 moisture sensor,
2 buzzer and vibrator have
been used. It has the detection
range up to 70cm.it does not
contain facilities, such as,
Particular object detection
facility, calling assistance and
GPS.
Sensing scheme:
The ATmega328 MCU uses
three types of sensors:
ultrasonic, water, Infrared
Receiver IR flame sensor and
buzzer. It can detect distance,
water and flame within an area
of 0.5cm to 200 cm. It provides
Calling, ODA and GPS facility.
3. “ Smart Walking Stick for
Blind Integrated with SOS
navigation system ”
Sensing scheme:
1 ultrasonic sensor with MCU
Raspberry Pi 3b
has been used. It provides
video calling only. NO GPS,
Google assistant and ODA
based particular object
detection is available.
Sensing scheme:
One buzzer and three different
types of sensors as Ultrasonic,
Water and IR Flame Sensor with
ATmega328 have been used. It
provides particular object
detection, Calling, ODA and
GPS facility.
4. “An Electronic Walking
Stick for Blinds”
Sensing scheme:
1 ultrasonic and 1 infrared
sensor with MCU ATmega328
have been used. It provides
only calling and message
facility via GSM. No facility
of particular object detection,
Google assistant, ODA and
GPS.
Sensing scheme:
One buzzer and three different
types of sensors as Ultrasonic,
Water, IR Flame Sensor with
ATmega328p MCU have been
used.
It can detect distance, water and
flame with in a zone of 0.5cm to
200 cm. It provides
particular object detection,
Calling, ODA and GPS facility.
Here, ultrasonic sensor has detected the object 0.5cm to
200cm correctly. After detecting an obstacle, the ultrasonic
sensor will produce loud sound through the buzzer by
which the blind person can easily cross the barrier. The IR
flame sensor is working properly as we have observed
from the experiments that the IR flame sensor is detecting

297
the flame from 0.2cm to 290cm exactly. Fig.7. shows the
flame detection through the IR flame sensor by blowing
both the LED at the time of fire sensing. The LED is also
glowing while detecting the flame as well as the buzzer is
used here to produce the medium sound to aware the
visually impaired Person.
Fig.7. Shows the flame detection in integrated module
The water sensor is detecting the water level from 0.2
cm to 3. 81cm.The output of the water sensor will
produce on the buzzer. So, the blind person can navigate
easily.Fig.8 shows all three sensors are in integrated
module.
Fig.8. All three sensors in integrated module
The water sensor will be helpful for the blind person to
detecting the presence of water or to protect him from
getting wet. It can also detect drain water if he put his stick
near to it. The observation data of all three sensors and
ODA application are tabulated in table 2.
TABLE.2. RESPONSE OF THE STICK AT VARIOUS
DISTANCES USING SENSORS AND ODA.
S.N
o.
Type of
Sensor/
ODA
Detect
Parame
ter
Min.Ra
nge
Max.Ra
nge
Output on
Buzzer
/Headphone
1
Ultrasonic
Sensor
HCSR-04
Object 0.5 cm 2 meter
Buzzer Loud
Sound
2
IR Flame
Sensor
Fire 0.2 cm
2.9
meter
Buzzer
Medium
Sound, LED
HIGH
3
Water
Sensor
Water 0.2 cm 3.81 cm
Buzzer
Light Sound
4
ODA
Applicatio
n
Every
Object
1 cm 50 meter
Audio on
Head-
phone
The Arduino UNO ATmega328 microcontroller has been
used here to control all the sensors. The object detection
application is also one of the advantages of this stick. This
application recognizes the object from 1 cm to 5000 cm
exactly. This is detecting the object with its name first and
then it is speaking the name of the object in voice form, the
blind person will hear the name of the object using a
wireless headphone. Fig.9 shows the detection and
reorganization of an object using ODA through which the
blind person can easily understand, what kind of object is
available before him.
Fig.9. Live demonstration of Object Detection Application (ODA)
Therefore, the visually impaired person can easily find the
equipment that he needed. The Google assistant feature is
also used here in this stick by using the android mobile. In
case of any critical situation, using this calling assistant
feature the blind person can make a call on the registered
mobile number or send the GPS location to relative. The
wireless headphone is used here to provide flexibility to the
visually disabled person and he can make a call easily. The
GPS location can be sent through what’s app or a simple
text message. Fig.10. shows the overall system of blind
person stick.
Fig.10. All Three Sensors and ODA Based Smart Blind Stick

298
The stick assists the blind human to recognize the object,
water and fire with the help of three different kind of
sensors. One android application ODA will help to detect
the object with name and the blind people can hear this
using Bluetooth head phone. The Google Assistant also
helps the blind person to call any relative. The select to
speak button at bottom corner of the mobile phone will help
the blind person to connect the internet easily.
VI. CONCLUSION
Our System should be highly recommended to the blind
person for safe navigation. The smart sensor-based object
recognition system for blind person has been
implemented successfully. This stick contains sensor
module to detect the object. This stick also provides the
lots of facilities to the blind person such as audio and
video calling, messaging and GPS Location sharing with
the relative in some critical situation through the use of
android mobile with Google assistant. Thus, our system
provides the navigation facility to the blind person in dual
mode, one from the sensor module and another from the
ODA and Google assistant module. If the blind person
wants to recognize the particular objects in his way than
he can use ODA application. The robustness of our
system is that it will work in almost every condition of
the environment. The future perspective of this system is
that the power can be given to the system by using the
solar panel. This system will also be used for smart
robotic system development. The cost can also reduce in
future perspectives.
ACKNOWLEDGEMENT
This work has been completed with the help and under
the supervision of my supervisor Dr. S.K. Shah. And
Mohd. Noushad. Without both of them guidance this
work cannot be feasible. Their great knowledge,
excitement and attention help me lots of to complete this
research. Mohd. Moushad and Dr. S.K.Shah assist me lot
during the sensors experiments. Ajay Prasad Nautiyal
helps me to use the software properly. So, I highly
thankful of the team member to accomplish this research.
REFERENCES
[1] Ghosh Pritha, Dey Naiwrita, De Rahul, et al, “Ultrasonic Sensor
Based Smart Blind Stick”, IEEE International Conference on Current
Trends toward Converging Technologies, Coimbatore, India, 29
November 2018.
[2] Himanshu Sharma, Meenakshi Tripathi, Amit Kumar, et al,
“Embedded Assistive Stick for Visually Impaired Persons “ , IEEE,
Bengaluru, India, 10-12 July 2018.
[3] Rout Subham, Mohapatra Saurav, Tripathi Varun, et al, “Smart
Walking Stick for Blind integrated with SOS Navigation System” ,
2nd International Conference on Trends in Electronics and
Informatics (ICOEI 2018) IEEE Conference Record: # 42666; IEEE
Xplore ISBN:978-1-5386-3570-4, Tirunelveli, India, 12 May 2018.
[4] Chaurasia Shashank and K.V.N. Kavitha, “AN ELECTRONIC
WALKING STICK FOR BLINDS”, IEEE, Chennai, India, 28
February 2014; 10.1109/ICICES.2014.7033988.
[5] Srinidhi Srinivasan and Rajesh M, “Smart Walking Stick” , IEEE,
Tirunelveli, India, 23-25 April 2019 ; 10.1109/ICOEI.2019.8862753.
[6] Shaha Apurv, Rewari Shubham and Gunasekharan Sankaradithyan,”
SWSVIP–Smart Walking Stick for the Visually Impaired People
using Low Latency Communication”, International Conference on
Smart City and Emerging Technology (ICSCET), 5 Jan.
2018;https://doi.org/10.1109/ICSCET.2018.853738510.1109/ICSCET
.2018.8537385.
[7] Chen Liang Bi, Su Jian Ping, Ming-Che Chen et al,” An
Implementation of an Intelligent Assistance System for Visually
Impaired/Blind People “ IEEE International Conference on Consumer
Electronics (ICCE),
2019;https://doi.org/10.1109/ICCE.2019.866194310.1109/ICCE.2019
.8661943.
[8] Mrs. S. Divya, Shubham Raj, M. Praveen Sha, “Smart Assistance
Navigational System for Visually Impaired Individuals”, IEEE
International Conference on Intelligent Techniques in Control,
Optimization and Signal Processing (INCOS), 2019 ;
10.1109/INCOS45849.2019.8951333.
[9] Kang Sung Jae, Ho YoungKim ,Moon In Hyuk,” Development of an
Intelligent Guide-Stick for the Blind”, IEEE International Conference
on Robotics and Automation (Cat. No.01CH37164), Proceedings
2001
ICRA;https://doi.org/10.1109/ROBOT.2001.93311210.1109/ROBOT.
2001.933112.
[10] Md. IkbalAllama, RahmanFaidur, KabirHasnatM.,” Microcontroller
Based Smart Walking Stick for Visually Impaired People”, 4th
International Conference on Electrical Engineering and Information &
Communication Technology (iCEEiCT), 13-15 Sept. 2018
;https://doi.org/10.1109/CEEICT.2018.862804810.1109/CEEICT.201
8.8628048.
[11] Daniel Charlebois, David G. Goodenoug, Stan Matwin,” Machine
Learning from Remote Sensing Analysis”, Proceedings of IGARSS
'93 - IEEE International Geoscience and Remote Sensing Symposium,
18-21 Aug. 1993;10.1109/IGARSS.1993.322515.
[12] MunirM. U.Hasnat, MahmoodFahad, Zeb Ayesha et al,” The Voice
Enabled Stick An Embedded Device for Active Guidance of Visually
Impaired Pedestrians”, 20th International Conference of Computer
and Information Technology (ICCIT), 2017;
10.1109/ICCITECHN.2017.8281844.
[13] ShrenikaR M., ChikmathSwatiS, Dr. KumarRavi A V et al,” Non-
Contact Water Level Monitoring System Implemented using
LabVIEW and Arduino”, 2017 International Conference on Recent
Advances in Electronics and Communication Technology
(ICRAECT), 16-17 March 2017; 10.1109/ICRAECT.2017.51.
[14] YaoXianhua, Zhang Ying mei, GaoLin et al,” Ground Water Level
Measurements using a Segmented Resistance Sensor”, 2009 IEEE
Instrumentation and Measurement Technology Conference, 5-7 May
2009;10.1109/IMTC.2009.5168667.
[15] SharmaShivani, ChandKailash, SharmaDeepak et al,” Development
of an early detection system for fire using Wireless Sensor Networks
and Arduino”, 2018 International Conference on Sustainable Energy,
Electronics, and Computing Systems (SEEMS), 26-27 Oct.
2018;10.1109/SEEMS.2018.8687333.
[16] Mozibul Hoque,Monir Hossain,Md. Yeasin Ali, et al,"BLIND
GUARD: Designing Android Apps for Visually Impaired
Persons",International Conference on Artificial Intelligence and
Computer Science Technology (ICAICST) 2021,29-30 June 2021.
[17] N. Loganathan, K. Lakshmi, N. Chandrasekaran, et al,"Smart Stick
for Blind People", 6th International Conference on Advanced
Computing and Communication Systems (ICACCS),6-7 March 2020.
[18] Kunja Bihari Swain, Rakesh Kumar Patnaik, Suchandra Pal et
al,"Arduino based automated STICK GUIDE for a visually impaired
person",IEEE International Conference on Smart Technologies and
Management for Computing, Communication, Controls, Energy and

299
Materials (ICSTM),2-4 Aug. 2017.
[19] Sung-Jae Kang, Young Ho, In Hyuk Moon,"Development of an
intelligent guide-stick for the blind", Proceedings 2001 ICRA. IEEE
International Conference on Robotics and Automation (Cat.
No.01CH37164),21-26 May 2001.
[20] Ahmed El-Koka,Gi-Hyun Hwang, Dae-Ki Kang,"Advanced
electronics based smart mobility aid for the visually impaired
society",2012 14th International Conference on Advanced
Communication Technology (ICACT),19-22 Feb. 2012.

Embedded_System_and_Smart_Phone_based_Object_Recognition_Technique_for_Visually_Impaired.pdf

More Related Content

Similar to Embedded_System_and_Smart_Phone_based_Object_Recognition_Technique_for_Visually_Impaired.pdf (20)

More from jaychoudhary37 (15)

Recently uploaded (20)

Embedded_System_and_Smart_Phone_based_Object_Recognition_Technique_for_Visually_Impaired.pdf