Text Extraction from Product Images Using State-of-the-Art Deep Learning Techniques
1 like933 views
The document discusses text extraction from product images, focusing on text detection and recognition through advanced machine learning techniques. It outlines model architectures, including VGG16 and CRNN-CTC, and describes the training process with a large synthetic dataset. Additionally, it covers the implementation of CTC loss for handling character alignment and addresses advanced techniques such as attention mechanisms.
Text Extraction from Product Images Using State-of-the-Art Deep Learning Techniques
- 1. 1
- 2. Text Extraction from Product Images Rajesh Shreedhar Bhat Data Scientist @Walmart Labs, Bengaluru MS CS @ASU, Kaggle Competitions Expert
- 3. Agenda ▪ Intro to Text Extraction ▪ Text Detection(TD) ▪ TD Model Architecture ▪ Training data generation ▪ Text Recognition(TR) training data preparation ▪ CRNN-CTC model for TR ▪ Receptive Fields ▪ CTC decoder and loss ▪ TR Training phase ▪ Other Advanced Techniques ▪ Questions ? 3
- 4. Introduction : Text Extraction 4
- 6. Image Segmentation - Input & Ground Truth 6
- 7. Ground Truth Label Generation 7
- 8. Text Detection – Model architecture ▪ VGG16 – BN as the backbone ▪ Model has skip connection in decoder part which is similar to U-Nets. ▪ Output : ▪ Region score ▪ Affinity score - grouping characters Ref:Baek, Youngmin, et al. "Character Region Awareness for Text detection." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. 8
- 9. Sample Output Region Score Affinity Score Region Score Affinity Score 9
- 12. Text Recognition – Training Data Preparation SynthText: image generation engine for building a large annotated dataset. 15 million images generated with different font styles, size, color & varying backgrounds using product descriptions + open source datasets Vocabulary: 92 characters Includes capital + small letters, numbers and special symbols 12
- 13. Text Recognition CRNN CTC model 13
- 14. CNN - Receptive Fields Usage of intermediate layer features in SSD’s in Object detection tasks. ▪ Receptive field is defined as the region in the input image/space that a particular CNN’s feature is looking at. 14
- 15. CNN features to LSTM Input Image (None, 128 * 32 * 1) Feature map (None, 1, 31, 512) Feature map (None, 1, 31, 512) SoftMax probabilities for every time step (i.e. 31), over the vocabulary. 15
- 16. Ground Truth and Output of TR task 16 Hello Input Image Ground Truth Output from LSTM model for 31 timesteps .. Time step t1 t2 t3 t4 t5 ... …. t27 t28 t29 t30 t31 Prediction H H H e e ... ... l o o o o Length of Ground truth is 5 which is not equal to length of prediction i.e 31
- 17. How to calculate the loss? NER model – loss: categorical cross entropy • Do we have labels for every time steps of LSTM model in CRNN setting ? • Can we use cross entropy loss? Answer is: NO!! 17
- 18. Mapping of Input to Output Corresponding Text Hello Corresponding Text Hello Can we manually align each character to its location in the audio/image? Yes!! But lot of manual effort is needed in creating training data. 18
- 19. CTC to rescue With just mapping from image to text and not worrying about alignment of each character to the location in input image, one should be able to train the network. Merge repeats Merge repeats Drop blank character 19
- 20. Connectionist Temporal Classification (CTC) Loss ▪ Ground truth for an image --> AB ▪ Vocabulary is { A, B, - } ▪ Let's say we have predictions for 3-time steps from LSTM network (SoftMax probabilities over vocabulary at t1, t2, t3) ▪ Given that we use CTC decode operation discussed earlier, in which scenarios we can say output from the model is correct?? 20
- 21. CTC loss continued .. t1 t2 t3 A B B A A B - A B A - B A B - • Merge repeats • Drop blank character AB t1 t2 t3 A 0.8 0.7 0.1 B 0.1 0.1 0.8 - 0.1 0.2 0.1 Ground Truth : AB Score for one path: AAB = (0.8 * 0.7 * 0.8) and similarly for other paths. Probability of getting GT AB: = P(ABB) + P(AAB) + P(-AB) + P(A-B) + P(AB-) Loss : - log( Probability of getting GT ) SoftMax probabilities 21
- 22. CTC loss perfect match t1 t2 t3 A - - - A - - - A - A A A A - A - A A A A • Merge repeats • Drop blank character t1 t2 t3 A 1 0 0 B 0 0 0 - 0 1 1 • SoftMax probabilities Score for one path: A- - = (1 * 1 * 1) and similarly for other paths. Probability of getting ground truth A: = P(A--) + P(-A-) + P(--A) + P(-AA) + P(AA-) + P(A-A) + P(AAA) Loss : - log( Probability of getting GT ) = 0 Ground Truth : A A 22
- 23. CTC loss perfect mismatch • Merge repeats • Drop blank character A t1 t2 t3 A 0 0 0 B 1 1 1 - 0 0 0 SoftMax probabilities Score for one path: A - - = (0 * 0 * 0) and similarly for other paths. Probability of getting ground truth A: = P(A--) + P(-A-) + P(--A) + P(-AA) + P(AA-) + P(A-A) + P(AAA) Loss : - log( Probability of getting GT ) = tends to infinity !! Ground Truth : A t1 t2 t3 A - - - A - - - A - A A A A - A - A A A A 23
- 24. Model Architecture & CTC loss in TF 24
- 25. Training Phase ▪ 15 million images ~ 690 GB when loaded into memory!! Given that on an average images are of the shape (128 * 32 * 3) and dtype is float32. ▪ Usage of Python Generators to load only single batch in memory. ▪ Reducing the training time by using workers, max_queue_size & multi-processing in .fit_generator in Keras. ▪ Training time ~ 2 hours for single epoch on single P100 GPU machine and prediction time ~1 sec for batch of 2048 images. 25
- 26. Other Advanced Techniques Attention - OCR Spatial Transformer Network – before text recognition Ref:Jaderberg, Max, Karen Simonyan, and Andrew Zisserman. "Spatial transformer networks." Advances in neural information processing systems. 2015. 26