Visual Translation Embedding Network for Visual Relation Detection (UPC Reading Group)
1 like351 views
This document summarizes a research paper on visual relation detection using a Visual Translation Embedding Network (VTransE). It introduces the tasks of visual relation detection and the challenges of existing joint and separate models. VTransE is described as mapping object and predicate features into a low-dimensional space, where relations can be modeled as vector translations. The document outlines VTransE's feature extraction method using classemes, locations, and bilinear interpolation of visual features. It evaluates VTransE on two datasets, finding that the embedding idea is effective and certain features improve relation detection and knowledge transfer between objects and predicates. Overall, VTransE performs comparably to state-of-the-art visual relation models.
![Visual Translation Embedding Network for Visual
Relation Detection
Slides by Fran Roldán
ReadAI Reading Group, UPC
20th March, 2017
Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang,
Tat-Seng Chua, [arxiv] (27 Feb 2017) [demo]](https://image.slidesharecdn.com/05visualtranslationembeddingnetworkforvisualrelationdetection-170320185136/85/Visual-Translation-Embedding-Network-for-Visual-Relation-Detection-UPC-Reading-Group-1-320.jpg)
Visual Translation Embedding Network for Visual Relation Detection (UPC Reading Group)
- 1. Visual Translation Embedding Network for Visual Relation Detection Slides by Fran Roldán ReadAI Reading Group, UPC 20th March, 2017 Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, Tat-Seng Chua, [arxiv] (27 Feb 2017) [demo]
- 2. Index 1. Visual Relation Detection 2. Visual Translation Embedding (VTransE) 3. VTransE Feature Extraction 4. VTransE Network 5. Evaluation 6. Conclusion 2
- 3. Visual Relation Detection ● Modeling and understanding the relationships between objects in a scene (i.e. “person ride bike”). ● Better generalization for other tasks such as image captioning or VQA. ● Visual relations are subject-predicate-object triplets, which we can model jointly or separately. 3
- 4. VTransE Translation Embedding ● For N objects and R predicates we have to learn: ○ Joint model: N2 R ○ Separate model: N+R. ● However, large appearance changes of predicate (i.e . predicate ride is different when object is bike than when the object is elephant). 4
- 5. VTransE Translation Embedding ● For N objects and R predicates we have to learn: ○ Joint model: N2 R ○ Separate model: N+R. ● However, large appearance changes of predicate (i.e . predicate ride is different when object is bike than when the object is elephant). ...is there any solution? 5
- 6. VTransE Translation Embedding ● Based on Translation Embeddings for representing large scale knowledge bases. ● Map the features of objects and predicates in a low-dimensional space, where relation triplet can be interpreted as a vector translation. We only need to learn the “ride” translation vector in the relation space. 6
- 7. VTransE Visual Translation Embedding Suppose are M-dim features of subject and object. We must learn a relation translation vector and the projection matrices . 7
- 8. VTransE Visual Translation Embedding Loss function to reward only deterministically accurate predicates: 8
- 9. VTransE Feature Extraction Knowledge Transfer in Relation ● Region proposal network (RPN) and a classification layer. ● Incorporation of knowledge transfer between objects and predicates, which can be transferred in a single forward/backward pass. ● Novel feature extraction layer: ○ Classeme (i.e. class probabilities). ○ Location (i.e. bounding boxes coordinates and scales). ○ RoI visual features (use of bilinear feature interpolation instead of RoI pooling). 9
- 10. VTransE Feature Extraction In order to extract we analyze three type of features: ● Classeme: N+1-dim vector of class probabilities (N classes and 1 background) obtained from object classification. ● Location: 4-dim vector such that: where are bounding boxes coordinates of subject and object respectively. ● Visual Features: D-dim vector transformed from a convolutional feature of the shape . 10
- 11. VTransE Feature Extraction Bilinear Interpolation Smooth function of two inputs: feature map F and an object bounding box. : X x Y grid split in box Since G is a linear function, V can be back-propagated to the bounding box coordinates 11
- 12. VTransE Optimization ● Multi-task loss function: ○ Object detection loss: ○ Relation detection loss: ● Loss trade-off: 12
- 13. VTransE Network Built upon an object detection module and incorporates the proposed feature extraction layer. 13
- 14. Evaluation Q1: Is the idea of embedding relations effective in the visual domain? Q2: What are the effects of the features in relation detection and knowledge transfer? Q3: How does the overall VTransE network perform compared to the other state-of-the-art visual relation models? 14
- 15. Evaluation ● Datasets: ○ Visual Relationship Dataset (VRD): 5,000 images with 100 object categories and 70 predicates. In total, VRD contains 37,993 relation annotations with 6,672 unique relations and 24.25 predicates per object category. ○ Visual Genome Version 1.2 (VG): 99,658 images with 200 object categories and 100 predicates, resulting in 1,174,692 relation annotations with 19,237 unique relations and 57 predicates per object category. 15
- 16. Evaluation (Q1) Q1: Is the idea of embedding relations effective in the visual domain? Isolate VTransE from object detection and perform the task of Predicate Prediction 16 R@K computes the fraction of times a true relation is predicted in the top K confident relation predictions in an image
- 17. Evaluation (Q2) Q2: What are the effects of the features in relation detection and knowledge transfer? 17 VRD VG
- 18. Evaluation (Q2) Q2: What are the effects of the features in relation detection and knowledge transfer? 18
- 19. Evaluation (Q3) Q3: How does the overall VTransE network perform compared to the other state-of-the-art visual relation models? 19
- 21. Conclusions ● Visual Relation task gives us a comprehensive scene understanding for connecting computer vision and natural language. ● VTransE designed to provide object detection and relation prediction simultaneously ● Novel feature extraction layer that enables object-relation knowledge transfer. 21