Segmentation - based Historical Handwritten Word Spotting using document-specific Local Features
Download as PPTX, PDF1 like1,114 views
This document discusses a segmentation-based approach for historical handwritten word spotting, introducing novel document-specific local features and a matching procedure that incorporates spatial context. The proposed method outperforms existing profile-based strategies and SIFT local features in effectiveness and efficiency, demonstrating superior performance through evaluations on specific datasets. Overall, the framework presents a significant improvement in identifying queried words in document images with varying writing styles and characteristics.
Segmentation - based Historical Handwritten Word Spotting using document-specific Local Features
- 1. SEGMENTATION-BASED HISTORICAL HANDWRITTEN WORD SPOTTING USING DOCUMENT-SPECIFIC LOCAL FEATURES KONSTANTINOS ZAGORIS1,2 IOANNIS PRATIKAKIS1 BASILIS GATOS2 Visual Computing Group Democritus University of Thrace Dept. of Electrical and Computer Engineering Xanthi, Greece National Centre of Scientific Research “Demokritos” Institute of Informatics and Telecommunications Athens, Greece 1 2
- 2. WHAT IS KEY WORD SPOTTING? • It is the task of identifying locations on a document image which have high probability to contain an instance of a queried word • without explicitly recognizing it. • It is related to Content-Based Image Retrieval systems. • Searching a word image from a set of unindexed document images using the image content as the only information source.
- 3. CURRENT LITERATURE TRENDS • Currently there are two distinct trends. (i) Segmentation-based and (ii) Segmentation-free approaches. • Their fundamental difference concerns the search space - segmented word images (segmentation-based) - complete document image (segmentation-free). We address the word spotting problem with a segmentation-based approach.
- 4. PREVIOUS LITERATURE Rath and Manmatha calculate two families of feature sets. • scalar type features that include aspect ratio, area, etc. • profile-based features that are based on horizontal and vertical words projections and the upper and lower word profiles. Zagoris et. al. created a similar set of profile-based features but: • encoded Discrete Cosine Transformation and • quantize through the Gustafson - Kessel fuzzy algorithm. Rodriguez and Perronnin extract features from a sliding window, based on the first gradient and inspired by the SIFT keypoint descriptor.
- 5. BAG-OF-VISUAL WORDS MODEL Recently, there was an influx of works based on the local features in the form of the Bag-of-Visual Words model. Llados et. al. evaluate the performance of various word descriptors : • a bag of visual words procedure (BoVW), • a pseudo-structural representation based on Loci Features, • a structural approach by using words as graphs, and • sequences of column features based on DTW. They found that the statistical approach of the BoVW produces the best results, although the memory requirements to store the descriptors are significant.
- 6. PROBLEMS WITH CURRENT LOCAL FEATURES Most works using local features are based on the Scale Invariant Feature Transform (SIFT) in order to describe the local information • The original application of these local features are the natural images which they have many structural differences compared to document images • The detection of the most powerful edges through pyramid scaling creates local points between text lines. • Invariant properties in the descriptor results in noise amplification so they are more sensitive to the noise and the complex texture of the background.
- 7. TEXTURE VS SHAPE FEATURES Features for word spotting which rely only on word shape characteristics are not effective in dealing with a document collection created by different writers, containing significant writing style variations. Although slant and skew preprocessing techniques can reduce the shape variations, they cannot eliminate the problem as the whole structure of the word is different in most of the cases. In this respect, we argue that although the shape information is meaningful, the texture information in a spatial context is more reliable.
- 8. DOCUMENT SPECIFIC LOCAL FEATURES (DSLF) Taking into account the aforementioned considerations, we propose: • novel local features which are specific for documents and a • matching procedure that does not rely on codebook creation (as on BoVW).
- 9. PROPOSED WORD SPOTTING FRAMEWORK Document Dataset Word Image Segmentation Word Image DSLF Calculation Matching Process Display Results Database DSLF Calculation ONLINE OFFLINE
- 10. KEYPOINT DETECTION AND SELECTION Keypoint Detection CCs Analysis Local Point Selection Gradient Orientation Quantization Convex Hull Corner Detection Entropy-based Final Keypoints Keypoint Filtering
- 11. KEYPOINT DETECTION AND SELECTION original document image orientation of the gradient vector quantization of the gradient vector orientation initial keypoints final keypoints
- 12. FEATURE EXTRACTION • The feature for the local keypoint is calculated upon the quantized gradient angles • An area of 18x18 pixels around the kP, is divided into 9 cells with size 6x6 for each of them. • Each cell is represented by a 3-bin histogram (each bin corresponds to a quantization level). • Each pixel accumulates a vote in the corresponding angle histogram bin. The strength of voting depends on the norm of the gradient vector and on the distance from the location of local point as shown at the following equation: x, y x, y x, y V s G 2 2 , 2 1 LP LP 3 9 2 x y x x y y s • The task of the 푠푥,푦 variable is to weigh the pixel participation to the histogram taking into account its distance from the kP.
- 13. MATCHING PROCEDURE • In the case of segmentation-based word spotting, the aim is to match the query keypoints to the corresponding keypoints of any word image in the document. • Local Proximity Nearest Neighbor (LPNN) search is implemented. • The advantage of LPNN search is two-fold: it enables a search in focused areas instead of searching in a brute force manner and it goes beyond the typical use of a descriptor by the incorporation of spatial context in the local search addressed.
- 14. MATCHING PROCEDURE Update the location for each keypoint to a new normalized space: ′ = 푝푥푖 푝푥푖 −푐푥 퐷푥 ′ = , 푝푦푖 푝푦푖 −푐푦 퐷푦 where: 푐푥, 푐푦 = 푘 푖=푥1 푖 푝푘 , 푘 푖=푦1 푖 푝푘 퐷푥 = 푘 푝푥푖 푖=1 − 푐푥 푘 , 퐷푦 = 푘 푝푦푖 푖=1 − 푐푦 푘 k denotes the total number of the keypoints in a word image.
- 15. EVALUATION - DATASETS BENTHAM DATASET • It consists of 50 high quality (approximately 3000 pixel width and 4000 pixel height) handwritten manuscripts written by Jeremy Bentham (1748- 1832). • The variation of the same word is extreme and involves writing style, font size, noise as well as their combination.
- 16. EVALUATION - DATASETS WASHINGTON DATASET • It consists of 20 document images from George Washington Collection of the Library of Congress • The documents are were scanned from microfilm in 300 dpi resolution.
- 17. EVALUATION STRATEGY • Two evaluation metrics: Precision at the k Top Retrieved words (P@k) and the Mean Average Precision (MAP). • P@5 is the precision at top 5 retrieved words. This metric defines how successfully the algorithms produce relevant results to the first 5 positions of the ranking list • MAP is a typical measure for the performance of information retrieval systems • For the experiments, the word image segmentation information is taken from the ground truth corpora. • The total word image queries for the Washington dataset was 1570 and for the Bentham dataset was 3668. • Both query sets contain words appearing in various frequencies and sizes • Evaluated against two previous segmentation-based profile-based strategies • Then, in order to highlight the advantage of the proposed DSLF, it was replaced by the SIFT but the proposed matching algorithm remained the same.
- 18. OVERALL PERFORMANCE EVALUATION RESULTS Washington 0.660 0.637 0.631 0.600 0.608 0.577 0.436 0.440 0.800 0.700 0.600 0.500 0.400 0.300 0.200 0.100 0.000 P@5 MAP WS CSPD SIFT Proposed Method Bentham 0.701 0.680 0.642 0.630 0.629 0.615 0.528 0.506 0.800 0.700 0.600 0.500 0.400 0.300 0.200 0.100 0.000 P@5 MAP WS CSPD SIFT Proposed Method
- 19. CONCLUSION In this work, novel local features are proposed driven by the challenges presented in historical handwritten word spotting scenarios. The proposed method outperformed both the profile-based strategies and the SIFT local features. Moreover, a matching procedure was presented based on Local Proximity Nearest Neighbour, that augments performance in terms of effectiveness and efficiency incorporating spatial context. The proposed framework achieves better performance after a consistent evaluation against two profile-based approaches as well as the proposed approach with the popular SIFT local features in two different handwritten datasets.
- 20. ΕΥΧΑΡΙΣΤΩ ΠΟΛY! THANK YOU!
Editor's Notes
- #5: Although there is an abundance of systems suitable for both modern and historical printed material, very few of these systems are suitable to handwritten documents due to noise sensitivity, character variation and text layout complexity:
- #7: we argue that it is not beneficial in document images to incorporate invariant properties in the descriptor of the local points as it results in noise amplification. We believe that the features that are invariant to rotation are more sensitive to the noise and the complex texture of the background.
- #8: Features for word spotting which rely only on word shape characteristics are not effective in dealing with a document collection created by different writers, containing significant writing style variations. Although slant and skew preprocessing techniques can reduce the shape variations, they cannot eliminate the problem as the whole structure of the word is different in most of the cases. In this respect, we argue that although the shape information is meaningful, the texture information in a spatial context is more reliable.
- #9: For the sake of clarity, it is worth to note that since the focus of this work is on features extraction and matching, the segmented word images used in the proposed approach are achieved from the available ground truth dataset without involving any particular word image segmentation method.
- #10: But initially let’s look the proposed word spotting framework
- #15: In the next stage, the LPNN for each keypoint that resides on the query image is addressed. LPNN is realized in a search area which is computed by taking into account a percentage (25%) of the already calculated distances Dx,Dy. During search, if there is one or more word keypoints in the proximity of the query keypoint under consideration, the Euclidean distance between their descriptors is calculated and the minimum distance is kept. This is repeated for each keypoint in the query image. The final similarity measure is the sum of all the minimal distances. If there is not a local point in its proximity then a penalty value is added to the similarity measure and it is equal to maximum Euclidean distance that can be calculated between the keypoint descriptors. As a final stage, the system presents to the user all the word images based on ascending sort order of the calculated similarity measure.
- #19: The proposed method outperformed both the profile-based strategies and the SIFT local features. It is worth to note, that the profile-based features were applied in words that were binarized, denoised, de-skew and de-slant as opposed to the local features that were applied to the original word images. Moreover, although the SIFT descriptor contains more information than the proposed local features (128 values against only 27), the latter performed better in both datasets