AUTOMATIC LICENSE PLATE RECOGNITION USING YOLOV4 AND TESSERACT OCR

https://iaeme.com/Home/journal/IJEET 58 editor@iaeme.com
International Journal of Electrical Engineering and Technology (IJEET)
Volume 12, Issue 5, May 2021, pp. 58-67, Article ID: IJEET_12_05_006
Available online at https://iaeme.com/Home/issue/IJEET?Volume=12&Issue=5
ISSN Print: 0976-6545 and ISSN Online: 0976-6553
DOI: 10.34218/IJEET.12.5.2021.006
© IAEME Publication Scopus Indexed
AUTOMATIC LICENSE PLATE RECOGNITION
USING YOLOV4 AND TESSERACT OCR
Adarsh Sai Daivansh Sham, Paritosh Pandey, Sambhav Jain and Dr. S. Kalaivani*
Department of Communication Engineering, School of Electronics Engineering,
Vellore Institute of Technology, VIT University, Vellore, Tamilnadu, India
*Corresponding Author
ABSTRACT
In modern times the quantity of on road vehicles is expanding very quickly. Most of
the time, it is important to verify the identity of these vehicles for authorization of the
transit regulation, overseeing parking garages. it is hard to check this colossal number
of moving vehicles physically. Subsequently, building up a precise automatic license
plate recognition model (ALPR) including character recognition is important to ease
the issues mentioned above. We have developed a model based on multiple types of
license plates from different countries. The dataset of images was trained using Yolov4
which uses CNN architectures. Character recognition was done using the Tesseract
OCR after multiple image pre-processing techniques and morphological
transformations. The proposed program has obtained an accuracy of 92% in license
plate detection and 81% in character recognition.
Keywords: recognition, accuracy, detection General Terms- Automatic License plate
recognition (ALPR), tesseract-OCR, image processing, Yolov4
Cite this Article: Adarsh Sai Daivansh Sham, Paritosh Pandey, Sambhav Jain,
S. Kalaivani, Automatic License Plate Recognition using Yolov4 and Tesseract OCR,
International Journal of Electrical Engineering and Technology (IJEET), 12(5), 2021,
pp. 58-67.
https://iaeme.com/Home/issue/IJEET?Volume=12&Issue=5
1. INTRODUCTION
The objective of ALPR is to separate the vehicle number from pictures of moving vehicles.
ALPR incorporates two significant steps; detecting the license plate region using bounding
boxes and recognition of the characters using image pre-processing techniques and tesseract-
OCR.
The paper intends to build up another and effective ALPR approach for multiple license
plates. The proposed approach is based on deep learning to solve plate detection and recognition
problems. Efficient CNN architectures are proposed in plate detection and recognition stages.
The CNN models depend on YOLO4 CNN design. The YOLO4 CNN design is altered to a

Automatic License Plate Recognition using Yolov4 and Tesseract OCR
shallow CNN design to distinguish and perceive little items (characters of tag), the quantity of
layers is little thought about with YOLOv4, which thus diminishes the running time. YOLO is
short for You Only Look Once. It is a real-time object recognition system that can recognize
multiple objects in a single frame. YOLO recognizes objects more precisely and faster than
other recognition systems. It can predict up to 9000 classes and even unseen classes. The real-
time recognition system will recognize multiple objects from an image and also make a
boundary box around the object. It can be easily trained and deployed in a production system.
YOLO is based on a single Convolutional Neural Network (CNN). The CNN divides an image
into regions and then it predicts the boundary boxes and probabilities for each region. It
simultaneously predicts multiple bounding boxes and probabilities for those classes. YOLO
sees the entire image during training and test time so it implicitly encodes contextual
information about classes as well as their appearance. Hence, facilitating the detection of the
license plate. The recognition of characters is done using the Tesseract OCR software after
image pre-processing techniques are done on the detected license plate, using Python language.
2. RELATED WORK
Computer vision and character recognition, algorithms for license plate recognition play an
important role in video analysis of the number plate image. Therefore they form the core
modules in any ALPR system. Nijhuis et al. [3] combined neural networks and fuzzy logic in
recognition of car number plate for the case of the Dutch number plates. ANN models was also
used for training and detection, along the character recognition using image pre-processing
techniques and Tesseract-OCR by Antonius Herusutopo et al.[5]. Moreover, CNN architectures
using YOLOv3 were implemented by Salah Alghyaline [6]. Sindh standard number plate
recognition is done by Quadri and Asif [7] wheregthe number plate region is cornered with the
help of yellow color identification, later using smearing algorithm the plate is segmented, then
the Optical Character Recognition (OCR) is used to identify the characters. However the type
of OCR algorithm used is not mentioned and the accuracy rate is also not given. Tejas et al. [8]
proposed Indian number plate detection and recognition using techniques like Sobel edge
detection, bounding box segmentation, and neural networks for recognition .
3. PROPOSED METHODOLOGY
The method used can be divided in 3 main phases:
• Firstly we have gathered a dataset of images containing cars and their respective license plate.
We have trained the dataset using YoloV4 which is based on a single Convolutional Neural
Network (CNN). The CNN divides an image into regions and then it predicts the boundary
boxes and probabilities for each region. In this case, we will train the dataset in order to
recognize the license plates and form bounding boxes around them. The weights obtained from
training the dataset is the converted to Tensorflow format for compatibility with python.
• Secondly, we have used image processing techniques, namely; grayscaling, Gaussian blur,
Otsu’s thresholding and binarization method being pre-processing techniques applied to the
detected license plate region, followed by morphological transformations and application of
contours around desired characters based on the dimensions of the characters and spatial
localization. This is done using OpenCV.
• Finally, the characters are segmented and recognition is done using the Tesseract-OCR.

Adarsh Sai Daivansh Sham, Paritosh Pandey, Sambhav Jain, S. Kalaivani
Figure 1. Proposed methodology Flow Diagram
3.1. Training Dataset Using Yolov4 and Detecting License Plate
Yolov4 is an object detection model. Object detection models are usually trained to look at an
image and search for a subset of object classes. These object classes are enclosed in a bounding
box and their class is identified. Yolov4 is a one-stage object detection model.
In contrast, a two stage detector uses a preliminary stage where regions of importance are
detected and then is classified to see if the object has been detected in these areas. The main
upside of a one stage detector is the speed it is able to make predictions quickly for real time
use[8].
Figure 2 Structure of One-stage detector (YoloV4)[8]
Backbone
The YoloV4 backbone architecture is made up of three parts:
• Bag of freebies ; they are set of methods that only increase the cost of training or change the
training strategy while leaving the cost of inference low. Some of those methods are data
augmentation, photometric distortion, geometric distortion, mix p augmentation and Cut mix.
• Bag of specials: Bag of special methods are the set of methods which increase inference cost
by a small amount but can significantly improve the accuracy of object detection. It consists of
mish activation function. Mish avoids saturation, which generally causes training to slow down
due to near-zero gradients drastically.

• CSPDarknet53: The Cross Stage Partial architecture is derived from the DenseNet architecture
which uses the previous input and concatenates it with the current input before moving into the
dense layer.
Neck (detector)
The main role of the neck is to collect feature maps from different stages.The structure of the
latter will consist of a Spatial Pyramid Pooling Layer which will allow to generate fixed size
features whatever the size of our feature maps
Figure 3. Structure for SPP layer
Head (detector)
The role of the head in the case of a one stage detector is to perform dense prediction. The dense
prediction is the final prediction which is composed of a vector containing the coordinates of
the predicted bounding box (center, height, width), the confidence score of the prediction and
the label which in our case, the bounding box will be around the license plate.
Figure 4. License plate detected in bounding box
As we see in Fig 4. the license plate has been detected with an accuracy of 92%.

3.2. Image Processing and Segmentation
The next phase after training the dataset of images and detecting the license plate is to apply
pre-processing techniques i.e. gray scaling, Gaussian smoothing, thresholding using Otsu’s
method.
Cropping the license plate from the bounding box
First step of the process is taking the bounding box coordinates from YOLOv4 detection phase
and simply taking the sub image region within the bounds of the box.
Figure 5. Resized image of license plate
Grayscaling
The importance of grayscaling is dimension reduction for example, in RGB images there are
three color channels and has three dimensions while grayscaled images are single dimensional.
Grayscaling also reduces the complexity of processing of the image On the other hand, the same
neural network will need only 100 input node for grayscaled images.
Applying Gaussian Smoothing
In Gaussian smoothing, every point of the input array is convolved using the Gaussian equation
as it is shown in equation 1 below. The output array is obtained by the summation of all such
points.
𝐺(𝑥) =
1
√2𝜋𝜎2
𝑒
−𝑥2
2𝜎2
(1)
In the case of an Image, a two-dimensional version of this function is used, which is just the
product of two one-dimensional functions. Mathematically, it can be expressed as:
𝐺(𝑥, 𝑦) =
1
2𝜋𝜎2 𝑒
−(𝑥2+𝑦2)
2𝜎2
(2)
where x & y are the distances from the origin in the horizontal axis and vertical axis
respectively and the standard deviation of the Gaussian distribution is denoted by σ.[12]
That results in a decrease of computational complexity when compared to its two-
dimensional counterpart
Thresholding and binarization using Otsu’s method
The image is then thresholded to white text with black background and has Otsu's method also
applied. This white text on black background helps to find contours of image.
Otsu’s binarization distinguishes the foreground from background and turns the latter black
as it is effective on bimodal images.
𝜎𝑤
2
(𝑡) = 𝑞1(𝑡)𝜎1
2
(𝑡) + 𝑞2(𝑡)𝜎2
2
(𝑡)
Where

𝑞1(𝑡) = ∑
𝑡
𝑖=1
𝑃(𝑖) 𝑞1(𝑡) = ∑
𝐼
𝑖=𝑡+1
𝑃(𝑖)
𝜇1(𝑡) = ∑
𝑡
𝑖=1
𝑖𝑃(𝑖)
𝑞1(𝑡)
𝜇2(𝑡) = ∑
𝐼
𝑖=𝑡+1
𝑖𝑃(𝑖)
𝑞2(𝑡)
𝜎1
2
(𝑡) = ∑
𝑡
𝑖=1
[𝑖 − 𝜇1(𝑡)]2
𝑃(𝑖)
𝑞1(𝑡)
𝜎2
2
(𝑡) = ∑
𝐼
𝑖=𝑡+1
[𝑖 − 𝜇1(𝑡)]2
𝑃(𝑖)
𝑞2(𝑡)
It finds a value of t which lies in between two peaks such that variances to both classes are
minimum.
For 2 classes, minimizing the intra-class variance is equivalent to maximizing inter-class
variance:
𝜎𝑏
2
(𝑡) = 𝜎2
− 𝜎𝑤
2
(𝑡) = 𝜔0(𝜇0 − 𝜇𝑇)2
+ 𝜔1(𝜇1 − 𝜇𝑇)2
= 𝜔0(𝑡)𝜔1(𝑡)[𝜇0(𝑡) − 𝜇1(𝑡)]2
which is expressed in terms of class probabilities 𝜔 and class means 𝜇, where the class
means 𝜇0(𝑡), 𝜇1(𝑡) and 𝜇𝑇 are:
𝜇0(𝑡) =
∑𝑡−1
𝑖=0 𝑖𝑝(𝑖)
𝜔0(𝑡)
𝜇1(𝑡) =
∑𝐿−1
𝑖=𝑡 𝑖𝑝(𝑖)
𝜔1(𝑡)
𝜇𝑇 = ∑
𝐿−1
𝑖=0
𝑖𝑝(𝑖)
Figure 6. Image after Otsu’s binarization
Morphological Transformations
The image then undergoes dilation using OpenCV in order to make contours more visible and
be picked up in future step.
Figure 7. Image after Dilation process
Application of contours and segmentation
We now use OpenCV properties in Python to apply contours in the form of rectangular boxes
around the characters and sort them left to right.

Figure 8. Contours applied in rectangular boxes form
Figure 9.Contours around desired charatcters
The individual characters of the license plate number are now the only regions of interest
left. We segment each sub image and apply a bitwise_not mask to flip the image to black text
on white background which Tesseract is more accurate with.
Finally we will apply a small median blur to eliminate any remaining noise.
Figure 10. Segmented characters of the image
3.3. Recognition of character using Tesseract OCR
Pre-processing techniques are required though for the accurate use of the tesseract-OCR.
It can be used to recognize both structured and unstructured data.
Figure 11. Structure of the Tesseract-OCR
This neural network architecture implements and combines feature extraction, sequence
modeling, and transcription into a unified framework. This model does not need character
segmentation. The CNN extracts features from the input image(text detected region). Thebdeep

bidirectional recurrent neural network predicts label sequence with some relation between the
characters.
Figure 12 Image with recognized characters
4. EXPERIMENTAL RESULT AND DISCUSSION
Images trained with Yolov4 using R-CNN and an input of 8000 iterations had a validation
accuracy of 98% and an error rate of less than 1.
Figure 13. Image with more unwanted texts in the license plate
License plate Recognition : 80%
Characters on license plate: V0DKAA
Characters read : V0DKAA
Character recognition was: 100%
We can see here that the license plate was smaller and full of unwanted texts.
Figure 14. Slightly noisy and blurry image from dashboard

License plate Recognition : 88%
Characters on license plate:KR696969
Characters read : KR696969
Character recognition was: 100%
4.1. Result using video
We can see below a screenshot from a video where license plate recognition of both of the cars
are done as they are moving , even when they are in the same frame.
Figure 15. Result from 1 frame of a video
As the video proceeds, the license plates and the characters are detected. It depends on the
frame they are at which moment.
Table 1. Accuracy of proposed ALPR model
ALPR Model License plate
recognition
Character
Recognition
Ours 98% 81%
[9] 85% 80
5. CONCLUSION AND FUTURE WORK
The program that has been developed here using YoloV4 to train images has had 98% validation
rate with an error rate of less than 1. This license plate detection model enables detection and
recognition of characters in different types of environments and on multiple types of license
plates. Pre-processing techniques have been used such as gray scaling, Gaussian smoothing,
Thresholding by Otsu’s method and other morphological transformations in order to make the
recognition of characters in the license plates easier. We have tested the program with further
30 samples of images and obtained 92% of accuracy in license plate detection and 81% of
accuracy in detection of characters. Our future works will be to enhance the character
recognition program by training individual characters so that the Tesseract-OCR would work
more efficiently.

REFERENCES
[1] J. A. G. Nijhuis, M. H. Ter Brugge, K. A. Helmholt, J. P. W. Pluim, L. Spaanenburg, R. S.
Venema and M. A. Westenberg. (1995) “Car
[2] license plate recognition with neural networks and fuzzy logic,” Proceedings of ICNN'95 -
International Conference on Neural Networks, Perth.
[3] Herusutopo, Antonius, et al. "Recognition Design of License Plate and Car Type Using
Tesseract Ocr and Emgucv." Communication and Information Technology Journal, vol. 6, no.
2, 2012, pp. 76-84
[4] Alghyaline, Salah. (2020). Real-time Jordanian license plate recognition using deep learning.
Journal of King Saud University - Computer and Information Sciences.
[5] Muhammad Tahir Qadri and Muhammad Asif. (2009) “Automatic Number Plate Recognition
System for Vehicle Identification Using Optical Character Recognition,” 2009 International
Conference on Education Technology and Computer, Singapore.
[6] K Tejas, K Ashok Reddy, D Pradeep Reddy, K P Bharath, R Karthik and M. R. Kumar, (2018)
“Efficient License Plate Recognition System with Smarter Interpretation Through IoT,” Bansal
J., Das K., Nagar A., Deep K., Ojha A. (eds) Soft Computing for Problem Solving. Advances
in Intelligent Systems and Computing, 817: 207-220.
[7] Md Yeasir Arafat, Anis Salwa Mohd Khairuddin, Uswah Khairuddin and Raveendran
Paramesran, (2019) “Systematic review on vehicular license plate recognition framework in
intelligent transport systems,” IET Intelligent Transport Systems.
[8] Bochkovskiy, Alexey & Wang, Chien-Yao & Liao, Hong-yuan. (2020). YOLOv4: Optimal
Speed and Accuracy of Object Detection.
[9] M M Shidore, and S P Narote. (2011) “Number Plate Recognition for Indian Vehicles”
International Journal of Computer Science and Network Security 11(2): 143-146.

AUTOMATIC LICENSE PLATE RECOGNITION USING YOLOV4 AND TESSERACT OCR

More Related Content

What's hot (16)

Similar to AUTOMATIC LICENSE PLATE RECOGNITION USING YOLOV4 AND TESSERACT OCR (20)

More from Angie Miller (20)

Recently uploaded (20)

AUTOMATIC LICENSE PLATE RECOGNITION USING YOLOV4 AND TESSERACT OCR