IRJET - Interaction Paradigm of IoT using Augmented Reality

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 07 Issue: 04 | Apr 2020 www.irjet.net p-ISSN: 2395-0072
© 2020, IRJET | Impact Factor value: 7.34 | ISO 9001:2008 Certified Journal | Page 2136
Interaction Paradigm of IoT using Augmented Reality
Siddharth Bhokare1
Full Stack Developer, Think Analytics, Mumbai, India
---------------------------------------------------------------------***----------------------------------------------------------------------
Abstract - IoT in other way could be a combination of
physical objects with virtual representations and services.
The foremost famous implementation of IOT involves Home
Automation Systems. The most recent formation of IOT is
now converging to IOE (Internet of Everything). Whileusing
this technology, one can access each and every device,
equipment and machinery within its surrounding through
any application or “middleware system”. Each home will
become a smart home in future, where user is going to be
having remote access to every and each appliance of his/her
house. For this technological enhancement The Augmented
Reality (AR) is among top trending technology of today i.e
creating visual solutions of digital problems and
experiencing and visualizing them within the world.
Augmented reality integrates the unique abilities of kinsfolk
that can't be replicated by AI, and bonafide concerns about
AI are raised by some visionaries.
AR only uses the camera of any smartphone and
therefore the application will display the digital information
associated with the environment ahead of the user by using
the phone camera. My Research’s goal is to combine the
powerful digitizing feature of Augmented Reality with IOT.
The Augmented Reality provides a way for presenting the
digital information over physical objects, using this
approach, we have got created a new interface paradigm for
IOT. The IOT device has its particular interfaces mapped on
the physical IOT device using AR. Instead of trying to appear
out the interface, user are visiting be just placing his
smartphone camera over the thing and interact with it
directly using AR interfaces. TheIOTdeviceisalsoconnected
with every other using digital wires through AR, using this
approach we are visiting connect inputs to a minimum of 1
or more outputs just by connecting digital I/Opointsthatwe
have got created in our project. This provides features like
sharing the feature of timer switch of oven with any device
like fan or bulb, or using fan regulator not just to control
speed of fan but also connect it digitally with light bulb and
handle the brightness/light intensity through the regulator
Key Words: Augmented Reality, (AR), Internet of Things,
IoT, tracking, interaction
1. INTRODUCTION
IoT in alternative method might be a mixture of physical
objects with virtual representations and services. The
foremost celebrated implementation of IOT involves Home
Automation Systems. The foremost recent formation of IOT
is currently connection to IOE (Internet of Everything).
Whereas victimisation this technology, onewill accessevery
and each device, instrumentality and machinery inside its
close through anyapplicationor“middlewaresystem”.Every
home can become a sensible aim future, wherever user goes
to be having remote access to each and every appliance of
his/her house. For this technological improvement the
increased Reality (AR) is among prime trending technology
of nowadays i.e making visual solutions of digital issues and
experiencing and visualizing them inside the globe.
Increased reality integrates the distinctive skills of ancestry
that cannot be replicated by AI, and bonafide issues
concerning AI square measure raised by some visionaries.
AR solely uses the camera of any smartphone and thus the
appliance can show the digital data related to the
atmosphere earlier than the user by victimisation the phone
camera. My Research’s goal is to mix the powerful digitizing
feature of increased Reality with IOT. The increased Reality
provides some way for presenting the digital data over
physical objects, victimisation this approach, we've got
created a replacement interface paradigm for IOT.
The IOT device has its specific interfaces mapped on the
physical IOT device victimisation AR. ratherthanattempting
to seem out the interface, user square measure visiting be
simply putting his smartphone camera over the issue and
move with it directly victimisation AR interfaces. The IOT
device is additionally connected with each alternative
victimisation digital wires through AR, victimisation this
approach we have a tendency to square measure visiting
connect inputs to a minimum of one or additional outputs
simply by connecting digital I/O points that we've got got
created in our project. This provides options likesharingthe
feature of timer switch of kitchen appliance with any device
like fan or bulb, or victimisation fan regulator not simply
management| to regulate| to manage speed of fan however
to attach it digitally with lightweight bulb and control the
brightness/light intensity through the regulator
2. AUGMENTED REALITY INSMARTENVIRONMENT
AR describes an integrated view where virtual objects are
blended into a true environment in real time (Milgram et al.,
1994). While the concept of enriching a true environment
with virtual objects goes back to the center of the 19th
centure. New commercial products with better display and
tracking functionalities like Microsoft’s HoloLens are
prototypes towards a future during which vision is
pervasively augmented with smart contact lenses or glasses
(Perlin, 2016). My approach is to integrating AR into smart
living environments . I aim to enhance the traditional view
with virtual interactive control elements to reinforce
interaction with the environment. Therefore depend on a

wise environment platform which has a minimum of a
twofold accessible interface.
3. TOOLS & TECHNOLOGIES
3.1 Software Tools
3.1.1 Unity 3DUnity:
Unity 3DUnity is also a cross-platform game engine
developed by Unity Technologies, that is in the main
accustomed produce laptop games and recreations for PCs,
consoles and cell phones. Among the primary placedeclared
just for OS X, at Apple's WorldwideDevelopersConferencein
2005, it's since been stretched to specialize in twenty seven
platform.
3.1.2. Vuforia SDKVuforia :
Vuforia SDKVuforia is associate degree increased Reality
computer code Development Kit (SDK) for cell phones that
empowers the formation of increased Realityapplications.It
utilizes laptop Vision innovation to understand and track
planate footage (Image Targets) and simple 3D objects, for
example , boxes, more and more. This image time period
ability empowers engineers to position and situate virtual
articles, for example, 3D models and different media, in
affiliation to real footage once these area unit seen through
the camera of a phone. The virtual question at purpose}
tracks the position and introduction of the image more and
more so read point .the point of view the perspective}onthe
protest compares with their point of view on the Image
Target, therefore it provides the idea that the virtual
question is also a locality of this gift reality scene..
3.2 Hardware Tools
3.2.1 Arduino Micro Controller:
Arduino is associate degree ASCII text file physics platform
supported easy-to-use hardware and computer code.
Arduino boards area unit able to scan inputs - light-weight
on a device, a finger on a button, or a Twitter message -
associate degreed switch it into an output. We are going to
tell your board what to undertake to by causation a gaggleof
directions to the microcontroller on theboard.Toundertake
to therefore we tend to use the Arduino artificial language
(based on Wiring), and thereforetheArduinocomputercode
(IDE), supported process.
3.2.2 ESP8266 Wi-Fi Module:
ESP8266[12] is also a system-on-a-chip (SoC) with
capabilities for 2 .4 gigahertz Wi-Fi (802.11 b/g/n,
supporting WPA/WPA2), general input/output (16 GPIO),
Inter-Integrated Circuit (I²C), analog-to-digital conversion
(10-bit ADC), Serial Peripheral Interface (SPI), I²S interfaces
interfaces with DMA (sharing pins with GPIO), UART (on
dedicated pins, plus a transmit-only UART are often enabled
on GPIO2).
Fig -1: ESP8266 Pin Configuration
4. IMPLEMENTATION
4.1 System Design
4.1.1 Hardware Design:
There are two major parts of Hardware Design. One is
creating an IoT device from micro-controller along with Wi-
Fi module ESP8266 which is used to access devices over
network connection. The Circuit Schematics of this is below
Fig -2: Enabling IoT using Arduino Micro-Controller
Second device is to control the AC device using that IoT
enabled micro-controller with relay. As per condition
switching pulses are passes from Arduino to relay device.
Where relay switch from normally open to normally closed
condition so at the output side, device get turnon,that’s why
relay is consider as bridging device between controller and
output AC devices. The Circuit Schematics of this is designs
given below
Fig -3: AC Device Controlling via Arduino
a) Arduino Flowchart
Arduino process is divided into two parts. One is controlling
of Input Devices and other is controlling of Output Devices.
Both process are being shown in below flowcharts.

Chart -1: Input points flowchart
Chart -2: Output points flowchart
4.1.2 Software Design
The Mobile Application has been developed using cross
platform development tool Unity 3D. The Designing part
includes design of tracker and objects thataremappedusing
Augmented Reality SDK. The controlling objects are placed
on tracker plane in Unity 3D as shown in figure below.
Fig -4: Objects Placement in Unity 3D environment
a) Unity 3D Program Flowchart
Mobile Application is developed using this platform. The
overall working process of the application is shown in
flowchart below
Chart -3: Mobile App Flowchart
4.1.3 PubNub Network
PubNub is nothing but a programmable network for
developing real-time applications; an evolution from three-
tier architecture, purpose built to handleall thecomplexities
of knowledge streams. PubNub operates at the sting of the
network to handle and apply logic to realtime data, thereby
minimizing latency to a hundred milliseconds or less
worldwide, and guaranteeing dependability and
quantifiability. PubNub is accustomed quickly push little

messages to one or a lot of devices (smartphones, tablets,
desktops, microcontrollers, etc.) basically, nearlyanydevice
which may create a TCP/IP association to net – what ismore
as back once more, for bi-directional communication
between devices. These messages is employed for human
communication (like on-line chat), machine-to-machine
management, IoT, geolocation, good homes, money
information, multiplayergames,andmucha lotof.PubNubis
aware of that security, information possession, and access
management area unit crucial to any realtime application
that is why we've engineered the entire information Stream
Network around these core tenets.
5. PROPOSED ALGORITHMS
Our projected work for this project is creating the
conception of digital wires. The digital wires are supported
AR I/O purpose objects and line renderer to connect the
Inputs with Outputs. Presentlybeforeshapingtheprocedure
for creating digital wires in IoT exploitation AR, there ar few
things that has got to be made public first. In Digital wire
conception, the IoT has been weakened into twomajorparts
considering the life-style interaction. In modus vivendi, we
tend to tend to maneuver with exclusively two styles of
electronic devices that are either Input or either Output.
Now exploitation this concept, we've broken IoT devices
usefulness in to 2 categories “Input purpose Devices” and
“Output purpose Devices”. Exploitation this we've created
the I/O points or Digital wire conception. Presently
considering this interaction, we've derived one major issue
i.e. “One single input can management multiple outputs but
one single output can exclusively be controlled byonesingle
input”. This statement is taken into consideration as back-
bone of our rule. Presently keeping this concept in mind,
we've designed the code rule to figure and alter
consequently thus creating the conception of digital wires
5.1 Algorithm for Working Implementation
The Algorithm is employed for writing script within the
Unity 3D for the backend working of Digital Wires.
5.1.1 Global Object Pseudo Code
STEP 1: Start
STEP 2: Create Global Variable of Dictionary (A key value
pair data type) in C# script
STEP 3: Create a worldwide variable of String Array
STEP 4: Create a worldwide accessible method with
arguments of the input name which is looking and also the
value which the input must provide
STEP 5: Then we've to test whether what quantity outputs
the precise input has been connected with using line
renderer
STEP 6: Then we put all the outputs the input object is
connected within the temporary string array
STEP 7: End
5.1.2 Output Object Pseudo Code
STEP 1: Start
STEP 2: Create a callback method to test for mouse input on
output object
STEP 3: Whenever the output object is clicked we check if
the output is previously connected withthealternativeinput
object or not by check the world Dictionary
STEP 4: If it's already there then we are visiting remove it
first from that connection
STEP 5: at the moment we are visiting put the output object
and also the input object within the Dictionary as Key Value
Pair
STEP 6: End
5.2 Algorithm for Network side Implementation
The Data is communicated between the software mobile
application and also the hardware IoT devices via the
PubNub network. The algorithm is employed on both
hardware side and software side of the system for
communicating with one another.
5.2.1 PubNub Subscribe Pseudo Code
STEP 1: Start
STEP 2: Create PubNub thread for continuous
communication with the PubNub network and for receiving
and sending Data
STEP 3: Using the PubNub thread for Global Object script to
send the encoded string
STEP 4: Create a separate script for PubNub Subscribe in
order to receive data
STEP 5: subscribe the actual channel on whichall devices are
sharing data
STEP 6: Check for the encoded string data received from the
network
STEP 7: Call the global object method for checking the
related outputs
STEP 8: End
5.2.2 PubNub Publish Pseudo Code
STEP 1: Start
STEP 2: Create object of PubNub Thread Class
STEP 3: Provide the Generated keys of Publishing and
Subscribing to thread object
STEP 4: Using PubNub thread subscribe to the particular
channel
STEP 5: Publish data whenever it is required by calling in
any script
STEP 6: End

5.3 Algorithm for Hardware Side Implementation
We have used Arduino micro controller to connect with the
network and intercommunicate with other nodes on the
network and control the devicesattachedphysicallywiththe
Arduino. Each input or output device attached with the
Arduino is acting as an IoT device and sharing or receiving
data from over the network via Arduino.
5.3.1 Input Points Pseudo Code
STEP 1: Start
STEP 2: Initialize Input object as digital input and assign Pin
Number STEP 3: In the Arduino loop method continuously
check for high and low input STEP 4: Whenever the value is
high we write the Button ID and High value and whenitslow
we write Low value line in order to serial line for Esp8266
STEP 5: For analogue inputs we first need to map it with
outputs levels
STEP 6: After then we have to write that Input ID along with
output value
STEP 7: End
5.3.2 Output Points Pseudo Code
STEP 1: Start
STEP 2: Initialize digital output or analogue output
accordingly and assign pin number
STEP 3: In the Arduino Loop method we need to run a loop
to check the receiving data stream from the network and
iterate over the stream to access the characters
STEP 4: In the stream, we need to check for encoded string
for Output ID and value
STEP 5: After checking we need to send that particular value
to the particular output
STEP 6: Clear the current stream and wait for anotheroneto
receive from the network
STEP 7: End
5.3.3 Networking Pseudo Code
STEP 1: Start
STEP 2: Make a connection with network using the previous
algorithm
STEP 3: Connect with PubNub using the Generated Keys for
Publish and Subscribe
STEP 4: Check if data is available and get the data stream
using PubNub object
STEP 5: Provide the Data Stream to outputAlgorithmifNode
is Output, keep in mind that we can only create one Arduino
either as input or either as output
STEP 6: For Input Node, use the PubNub object to publish
the data of Input Points Algorithm
STEP 7: End
Fig -5: Room with multiple virtual control elements
Fig -6: Control element for the roller shutter item type
Fig -7: Scanning of a QR code using the ARGI app.
6. RELATED WORK
There are many other works which aim to implement a
distant control for a wise living environment.Asanexample,
Seifried et al. presented the CRISTAL tabletop system to
manage devices during a lounge withgestures(Seifriedetal.,
2009). While such systems provide a central remote, they'll
not use in free space but are sure to a particular location.. As
compared, this AR-empowered approach grants more
possibilities to visualise the interaction capabilities of a tool
and visual feedback. Although, this admit that a user must
wear an AR device rather just like the HoloLens to be ready
to use our system which could get uncomfortable the longer
a user wears the device. The interaction with smart
environments for a more robustassistanceinadulthoodisn't
among the standard application domains of AR like medical

visualization, maintenance and repair, or teaching and
learning (Azuma, 1997)(Wu etal.,2013).Though,Kriestenat
al. purposed an AR approach to interact with a wise
environment employing a mobile which is analogoustoours
(Kriesten et al., 2010). However, my approach goes a step
further because it enables free space gestures with the
assistance of a head-mounted AR device instead of a mobile,
Also ARGI relies on Eclipse Smart Home as a proved & freely
available smart environment solution which is useful to
extendability and reliability
7. CONCLUSION
In future, machines alone may be dominatedecision-making
in most examples. There will nevertheless be a fairly long
interim during which the foremost intellectual advances are
made by kinsfolk and computers working together in
intimate association. IoT is such a section that can integrate
human and computer Intelligence to unravel emerging
problems. Both affect the longer term of work,
organizational productivity, the blurring of industry
boundaries, and other legal, policy, and governance tasks.
For the nonce, appropriate IoT designs should maintain a
reasonable level of human control and oversight and
provides mankind an opportunity to urge familiar with
delegating control to machines.
This paper presents the use of augmented reality within the
sphere of home automation. AR enhances the control of the
devices and appliances withtheusers operatingthem.Nowa
days it's possible to talk and control the appliances remotely
with the emergence of smart devices. Many methods are
accustomed control the appliances remotely but it becomes
difficult for the users to use the smart devices because of the
GUI(Graphical User Interface) control. For this solution,
augmented reality (AR) technology is employed for simple
operation of home appliances connected by home networks.
AR technology can provide virtual graphics yet because it
apply additional information to defined areas via camera
displays. To create it more popular among people the AR
concept should be made cheaper and affordable for folk.
REFERENCES
[1] David, Nathan, Abafor Chima, Aronu Ugochukwu, and
Edoga Obinna. "Design of a home automation system
using arduino." International Journal of Scientific &
Engineering Research 6, no. 6 (2015): 795-
801,[Accessed: October 14, 2017]
[2] Mehmet Kesim, Yasin Ozarslan, Augmented Reality in
Education: Current Technologies and the Potential for
Education, In Procedia - Social and Behavioral Sciences,
Volume 47, 2012, [Accessed: October 14, 2017]
[3] Caudell, Thomas P., and David W. Mizell. "Augmented
reality: An application of heads-updisplaytechnologyto
manual manufacturing processes." In System Sciences,
1992. Proceedings of the Twenty-Fifth Hawaii
International Conference on, vol. 2, pp. 659-669. IEEE,
1992, [Accessed: October 14, 2017]
[4] Unity 3D, https://unity3d.com/ , [Accessed:October14,
2017].
[5] Vuforia,https://developer.vuforia.com/,[Accessed:
October 15, 2017]
[6] https://www.pubnub.com/, [Accessed: October 15,
2017].
[7] https://www.arduino.cc/en/Guide/Environment,
[Accessed: October 15, 2017
[8] Hao, Li, Xue Lei, Zhu Yan, and Yang ChunLi. "The
application and Implementation research of Smart City
in China." In System Science and Engineering (ICSSE),
2012 International Conference on, pp. 288-292. IEEE,
2012, [Accessed: October 14, 2017]Un.org, “United
Nations”, http:// http://www.un.org/, [Accessed:
October 14, 2017].

IRJET - Interaction Paradigm of IoT using Augmented Reality

More Related Content

What's hot (20)

Similar to IRJET - Interaction Paradigm of IoT using Augmented Reality (20)

More from IRJET Journal (20)

Recently uploaded (20)

IRJET - Interaction Paradigm of IoT using Augmented Reality