Thai License Plate Recognition using SSD MobileNet and EasyOCR

In this paper, we propose Thai license plate recognition for video utilizing deep learning approaches. Although many methods were proposed in the literature, most of them evaluated their performance using images not videos. Performing license plate recognition on videos is more challenging than that of images because moving cars cause blurred images, as well as incorrect and redundant number plates will be obtained. Therefore, our goal here is to automatically save the correct vehicle registration number with less redundancies in a database. To do so, we mainly use the SSD MobileNetV2 for license plate detection from video frames and then utilize EasyOCR for number plate recognition. Moreover, we introduce the post-processing process to filter out redundant and incorrect number plates before saving the number plates into the database. The experimental result of 100 images demonstrates that SSD MobileNetV2 obtains 99.00% accuracy for license plate detection, while EasyOCR achieves 92.00%-character accuracy for number plate recognition. The evaluation on videos shows that our proposed procedure achieves the license plate detection accuracy of 91.70%, the number plate recognition accuracy of 83.46%, and the character recognition accuracy of 96.99%.


INTRODUCTION
Nowadays, License Plate Recognition (LPR) plays a crucial role in many applications such as parking system management [1], automatic entry gate [2], traffic monitoring system [3,4], and vehicle tracking and identification [5,6].The main purpose of LPR is to read a license plate image and transform the optical data into digital information.To accomplish this work, there are three main steps, including license plate detection, character segmentation, and character identification.Several approaches were proposed to deal with this task.However, the types (i.e., size, shape, format, and layout) of license plates vary from each country and the characters of each country are also different.Therefore, the models of license plate detection and character recognition for each country need to be specifically trained and evaluated.Even for the same country, the types of license plates also vary.In this paper, we are interested in Thai license plate recognition with two lines, where the first upper line is the number plate, and the second lower line is the province name.However, we here consider only number plate identification.There are several approaches in the literature for Thai license plate recognition.We review some approaches related to our work as follows.
Thaiparnit et al. [6] proposed a tracking system based on license plate recognition.They extracted the license plate from the middle of the image by checking the number, and then applied k-Nearest Neighbor to recognize numbers on a license plate.It was able to recognize license plates from the tilt position at an angle of 80 degrees and an angle of 100 degrees for distances greater than 1 meter with 50% accuracy from the samples of 30 images.Puarungroj et al. [7] focused on Thai motorcycle license plates.They used Convolution Neural Network (i.e., MobileNets and Inception-v3) to detect license plates as well as number plate identification.The result of character recognition using MobileNet on 120 images, which contained 886 characters, was 91.76%.Sirawit et al. [2] developed Thai license plate recognition for an automatic barrier gate.They applied a multilayer perceptron (MLP) and a support vector machine (SVM) for character recognition.The results showed that the error rate of MLP is 0% and the error rate of SVM is 15%.Kitvimonrat et al. [8] focused on solving the problem of distorted license plate images by rotating the plate image before the recognition process.They used a Single Shot Detector (SSD) with a MobileNet for license plate detection.Then, they proposed a convolutional neural network (CNN) model for character recognition.They showed through the experimental results that their proposed method provided a high accuracy rate when license plates are distorted compared to their method with no rotation process.Sasakorn et al. [9] proposed Thai license plate detection and recognition based MRCNN and YOLO.The overall accuracy of their system is 89.08% including one letter wrong.
Some license plate recognition methods recently proposed for other countries are as follows.Wang et al. [10] proposed Korean license plate recognition for dealing with the new types of license plates in South Korea.They applied deep neural networks for both license plate detection and number plate recognition processes with the recognition accuracy of 98.94% using 200 images collected from the parking lot.Awalgaonkar et al. [11] proposed a license plate recognition system in India, while the characters in the license plate are in English.They used SSD with MobileNetV1 for license plate detection and applied EasyOCR and Tesseract OCR for number plate recognition.They obtained 94.87% accuracy for license plate detection using 156 images.Their experimental results also showed that the EasyOCR performed better than the Tesseract OCR.
Although there are various approaches in the literature, most of them evaluate license plate recognition based on images.However, in the real-world applications, the input of the system should be video and capturing images from video with the car moves causes blurred images resulting in the recognition rate lower.Moreover, deep learning-based approaches have recently gained popularity for solving this task due to its high accuracy prediction.Therefore, in this paper, we are interested in Thai license plate recognition from video using deep neural networks-based methods.Specifically, we propose to apply an SSD with MobileNetV2 for license plate detection and EasyOCR for optical character recognition.Our objective is to automatically capture the car images from a video, translate the optical character in the license plate into the digital characters, and save the license plate number into a database.These are to access the vehicle information quickly and easily through an application.Our contribution in this work is a new framework for Thai license plate recognition system based on the state-of-the-art deep learning convolution neural networks, composed of 3 steps: 1) license plate detection, 2) number plate identification, and 3) saving number plate in a database.We then did the experiments to demonstrate that the proposed framework is promising.

RELATED WORKS 2.1 Single Shot MultiBox Detector
Single Shot MultiBox Detector (SSD) [12] is a method for detecting objects in an image using a single deep neural network.It is fast in speed and high-accuracy object detection algorithm.SSD has a base VGG-16 network, which is used for feature extraction, followed by 6 more auxiliary convolution layers for object detection.SSD detects 8,732 default boxes of different scales, shapes, and aspect ratio on different output layers for a better coverage of location, scale, and aspect ratios.After that SSD drops predictions that have a confidence score that is lower than 0.01.Non-maximum suppression (NMS), which is a method to remove incorrect detection results, is then applied to remove duplicate predictions pointing to the same object and keep the top 200 detections per image.The input of the SSD is an image with ground truth boxes for each object during training.Data augmentation is also used to make the model more robust to various input object sizes and shapes.SSD can work together with other image recognition models (e.g., Inception-v3, MobileNets) to detect multiple objects bounding boxes and recognize their categories at the same time.

MobileNetV2
MobileNetV2 [13] is a fast mobile-based convolutional neural network architecture developed by Google, which was extended from MobileNets [14].It provides real-time classification capabilities under computational constraints on devices such as smartphones.The model was trained from ImageNet.However, we can transfer this pre-trained model to our dataset.The MobileNetV2 architecture has two main components.The first one is an inverted residual block, and the second one is a bottleneck residual block.MobileNetV2 is a major advancement over MobileNetV1 in terms of classification, object identification, and semantic segmentation for mobile visual recognition.MobileNetV2 is available as part of the TensorFlow-Slim Image Classification Library, and the installation and the usage can be found at [20].In this work, we apply MobileNetV2 instead of VGG16 to SSD for license plate detection.

EasyOCR
OCR is formerly known as Optical Character Recognition, where its goal is to detect the texts from images or documents and convert them into the normal editable text.EasyOCR is one of the OCR libraries like any other OCR e.g., tesseract of Google.It is a python package that supports 80+ languages for optical character recognition [15].It was created by the company named Jaided AI Company.EasyOCR detects characters in an image using the CRAFT algorithm [17].The recognition model is a CRNN [18], which is composed of three main components: feature extraction using ResNet, sequence labeling with LSTM (Long Short-Term Memory), and decoding based on CTC (Connectionist Temporal Classification).Given an input image, EasyOCR provides the output as the bounding boxes of detected words, text and confidence of each word, where the format of each bounding box is [  ,   ,   ,   ], and the high value of text's confidence refers to the high accuracy of each text.The installation and the usage of EasyOCR is at [16].

PROPOSED THAI LICENSE PLATE RECOGNITION FRAMEWORK
In this section, we describe our proposed Thai license plate recognition framework.Given a video as input, our purpose here is to automatically save the text of the number plate in a database, where we are interested in the car license plate.The proposed method consists of three main processes, including 1) license plate detection, 2) number plate recognition, and 3) saving the number plate in the database.The details of each process are explained in each following subsection.

License plate detection
License plate detection is the process of locating the area of license plate from the input image.Our license plate detection process is shown in Figure 1.We use the pre-trained MobileNetV2 model for feature extraction, where this pre-trained model was learned using COCO dataset [19].Then, the SSD is used for license plate detection.We re-trained the SSD model by using our own dataset, where the

Number plate recognition
After we obtain an image of the license plate, we utilize the Easy-OCR library for text recognition.We selected the EasyOCR because it supports Thai language, and its performance is better than another popular library like the Tesseract OCR [11].Especially for Thai characters, the Tesseract OCR does not perform well [22].The input of this process is an image of the license plate (LP), while the result is the text of the number plate.The results of EasyOCR consist of bounding boxes of detected words, text, and confidence of each word.However, the results may not be satisfied as we may obtain more than one bounding box indicating the region of the number plate.We here introduce post-processing to solve the problems as Figure 2.There are three main problems that we must solve to obtain the correct text on the number plate.The first problem is where we got more than one bounding box of plate numbers, including the region of the province, and other regions.In this case, we select a bounding box that the height is over 1/3 of the plate image as the region of plate number.The second problem is where we got two bounding boxes as the region of the number plate.To deal with this case, we concatenate these two bounding boxes together.
The third problem is where the tilting of a license plate is large, resulting in EasyOCR finding the text of digit before the text of character.Thus, the result after concatenating two bounding boxes is as "9999AB", for example.Thus, we swap these words to obtain the result of "AB9999".To do so, we compare the obtained text of plate number with the regular expression of "\D{1,2}\d\{1,4}"; if the plate number does not match this pattern, then we swap them.

Saving number plate in database
The results of plate image obtained from a video, even though we lower the frame rate to 5 frames per second, are duplicate license plates, while we do not need to save the duplicate text of a number plate in the database.In addition, the number plate after the recognition process may be incorrect, such as only two characters or only four digits are recognized.Thus, we need to filter out these two problems.Our process is shown in Figure 3.To solve the duplicate results, we calculate the similarity of the current number plate with the previous one; if they are 100% similarity, we do not save the current one into the database.For the second problem, we apply the regular expression (Regex) to filter out the incorrect pattern of the number plate, where our pattern of number plate is "\D{1,2}\d\{1,4}".We found through the experiments that the results after this process are proper.

EXPERIMENTAL RESULTS
In this section, we demonstrate the performance of the proposed framework using both images and videos.As there is no Thai license plate image dataset and videos available on the Internet, we here collect car images and videos ourselves.More details about our images and videos are explained below in each subsection.We first perform the proposed method on an image dataset.This is to see the performance of SSD MobileNetV2 and EasyOCR for license plate detection and character recognition, respectively.Next, we apply the proposed method on videos to confirm that our proposed system is promising to automatically save the number plates in a database.As SSD MobileNetV2 may return more than one bounding box as the location of license plate, we set the threshold of confidence value is at 0.8 for selection of the license plate region.We consider three types of accuracy to evaluate our proposed method, namely the detection rate, the number plate recognition rate, and the character recognition rate.Detection rate depicts the performance of SSD MobileNetV2 for license plate detection.Recognition rate demonstrates the percentage of the number plate that correctly predicted all digits and characters of the number plate.Notice that if the proposed method incorrectly predicts only one character, it means that the number plate is an incorrect prediction.The character recognition rate demonstrates the percentage of characters that are correctly predicted compared to all characters retrieved from all license plates.

Image dataset
In the first experiment, we evaluated the performance of the retrained SSD MobileNetV2 and EasyOCR using our image dataset.The images in our dataset are all different cars, consisting of images in a different resolution, light condition, size of license plate, as well as license plate tilt.We use 400 images for re-training SSD MobileNetV2 and another 100 images for evaluation.The summarized results are shown in Table 1, showing the number of our testing images and three types of accuracy: detection rate, number plate recognition rate, and character recognition rate.It can be seen from the results that the re-trained SSD MobileNetV2 achieved 99% accuracy detection (correctly detected 99 license plates from 100 testing images).The failure depicts that our SSD MobileNetV2 returned the bounding box of its license plate, but its confidence value is very low (e.g., 0.127 which is lower than 0.8).So, we do not consider this bounding box as a region of license plate.We then evaluated the performance of EasyOCR with post-processing using 99 extracted license plate images.The results indicate that Easy-OCR achieved 72% and 92% accuracy of number plate recognition rate and character recognition rate, respectively.More details on license plate recognition results are shown in Table 2, indicating that EasyOCR with post-processing misread only one character, two characters, and more than two characters as 19 images, 5 images, and 3 images, respectively.In other words, most of them misread only one character of a number plate.We also compared EasyOCR to Tesseract OCR.The results in Table 1 and Table Table 2 show that the EasyOCR significantly outperforms Tesseract OCR.

Video Dataset
In the second experiment, we evaluate the proposed method using videos.We collected videos at an entry of Prince of Songkla University around 7.00-8.00am. and 16.00-17.30pm. for 5 days.Therefore, we got 10 videos, each of which is around 4 to 20 minutes long.The total number of cars is 277.The number of cars, detection rate, number plate recognition rate, and character classification rate of each video are shown in Table 3.The results show that our system correctly detects 254 license plates out of 277 cars.From 254 license plate images, the system obtains 83.46% and 96.99% of the number plate recognition and the character recognition, respectively.We observe through the experimental results that the number of incorrect license plate recognition is 42 license plates, and 39 out of 42 these license plates are incorrectly predicted with only one character.Notice that the same license plate may have duplicate data in the database caused by incorrect number plate recognition.However, if we found one of the duplicate number plates is correctly predicted, we consider the recognition result of that number plate to be correct.

Discussion and application
In this subsection, we describe the factors that affect our results.First is the angle of capture should not be larger than 45 degrees.Second, the resolution of the detected license plate should be between 170x80 and 1200x600, otherwise our detection model may In our experiment, we use a smartphone and a camera with low performance for recording our videos, so that they do not fit for a moving car.This may infer that if a camera can capture a moving car well, the detection rate will be better.Moreover, the blurred image will be reduced.Consequently, the error rate of character recognition should be lower.Fourth, our system did not perform well on unclear number plates caused by its dirty or damaged.This problem will be considered in the future.To utilize our Thai license plate recognition framework, an example of GUI at the checkpoint is shown in Figure 4, and GUI for accessing car information is illustrate in Figure 5.

CONCLUSIONS
In this paper, we propose a Thai license plate recognition framework for online video.Our goal is to automatically save the correct license plate number with less redundancies in a database.We utilize the-state-of-the art deep learning approaches namely the SSD MobileNetV2 and the EasyOCR library for license plate detection and number plate recognition, respectively.Our proposed framework consists of three main steps.The first step is to detect license Therefore, in our opinion, it works for online applications.The failure of license plate detection is mostly caused by bad input images such as blurry images.Otherwise, our re-trained SSD MoblieNetV2 performs well.The EasyOCR is also a good library for reading Thai characters from an image.It overall correctly reads the characters from the license plate image.In most cases of incorrect prediction, only one character is incorrectly predicted.However, we will improve its accuracy in the future.Moreover, EasyOCR works for reading the province name as well if the region of province name has enough resolution.Thus, we will include the province name recognition into our system in the future.

Figure 2 :
Figure 2: Number plate recognition process

Figure 4 :
Figure 4: Example of GUI at Checkpoint of an Entry

Figure 5 :
Figure 5: Example of GUI for accessing car information

Table 1 :
License Plate Detection and Recognition Results Number of testing images Detection Acc.Number plate Recog.Acc.Char.Recog.Acc.

Table 2 :
The number of license plates that misread 0 to more than 2 characters

Table 3 :
Results of 10 videosVideosNumber of cars Correct plate detection Correct number-plate rec.Correct character rec.
not detect license plates.Bear in mind that the performance of the detection model depends on the image dataset used during training the model.Thus, we recommend collecting some training images from the real environment.Third, a car should stop at the checkpoint.If not, the grabbed image may be blurred.The most failure of license plate detection in our experiments is caused by this problem.