Making methods with vision APIs, online data & network building (lessons learnt)
0 likes170 views
The presentation by Dr. Janna Joceli Omena discusses developing digital visual methods using computer vision APIs for social and medium research. It outlines a computer vision network approach to analyze image collections by examining content, audience, and circulation, while emphasizing the creation of reproducible methodologies and software tools for data manipulation. Key findings highlight the importance of human intervention in method implementation and the need to understand the limitations of vision APIs.
![7
Chao, T. H. J. (2021). Memespector GUI: Graphical User
Interface Client for Computer Vision APIs (Version 0.2)
[Software]. Available from https://github.com/jason-chao/
memespector-gui.
Memespector GUI](https://image.slidesharecdn.com/caisclosingpresentation-makingmethodspdf-220211144150/95/Making-methods-with-vision-APIs-online-data-network-building-lessons-learnt-7-638.jpg)
![Offline Image Query and Extraction Tool
8
Chao, T. H. J. & Omena, J. J. (2021). Of
fl
ine Image Query and Extraction Tool
(Version 0.1) [Software]. Available from https://github.com/jason-chao/of
fl
ine-
image-query.](https://image.slidesharecdn.com/caisclosingpresentation-makingmethodspdf-220211144150/95/Making-methods-with-vision-APIs-online-data-network-building-lessons-learnt-8-638.jpg)
![10
The method recipe
The method protocol
analyse
images
QUERY OR QUERIES DESIGN
RESEARCH QUESTION
DATASET
DESIGN
PROCESS
VISUALISATION
PROCESS
VISUAL
NETWORK
ANALYSIS
images URL, engagement metrics, timestamps .csv
EXTRACT IMAGE METADATA
IMAGES METADATA
alike Image Tagnet Explorer, Tumbrl Tool, Google Image Extractor
according to analysed the digital platform
DIGITAL PLATFORM
Alike Instragam, Tumbrl, Google Image ]
IMAGES URL
.csv, .tsv, .txt file with URLs
ORGANISE AND CLEAN IMAGE METADATA
alike Excel, Google Spreadsheet
DOWNLOAD IMAGES FROM URL
DownThemAll or similar
RESIZE IMAGES
BulkResize or similar]
FOLDER OF IMAGES
[.native format]
FOLDER OF RESIZED IMAGES
[.native format]
alike Google Vision API
USE THE WEB VISION API SERVICE
CREATE AN API KEY
BUILD OR USE AN ALREADY EXSISTING SCRIPT
script (.py or .php) es: “from google.cloud import vision (...)
NETWORK
[.gexf]
.csv
VISION API METADATA
image url label web entitites image domain
BUILD THE EDGES/NODES TABLE
BUILD THE NETWORK
Table2Net or manually]
VISUALISE THE NETWORK
Gephi]
Gephi image preview plugin]
*Image Network Plotter Script works as well on Pyhton]
INSTALL THE GEPHI
IMAGE PREVIEW PLUGIN
Gephi + domain knowledge]
[es: ForceAtlas2 Spatialisation]
VISUALISE AND SPATIALIZE IT ROUGHLY
CHANGE NETWORK APPEREANCE
ACCORDING TO DATA
ADD PICTURES TO THE NETWORK
label
VISUALLY ANALYSE THE NETWORK
PRESENT/STAGE THE NETWORK
label
GO BACK TO THE
RESEARCH QUESTIONS
label
label
*size
*color
[attributes]
[in-degree, degree,out-degree]
EXPORT THE NETWORK IN AN VECTOR FORMAT
EDIT
ADD
ANNOTATIONS
ADD
A TITLE
ADD
KEY
REFINE
COLORS
ADJUST
LABEL SIZE
[.svg, .eps]
Adobe Illustrator, Inkscape]
mediums / software
technical practice
output
researcher intervention
VISUALISATION
PRESENTATION
label
label
annotation
Title
color 1
color 2
GEPHI DATA LAB SPREADSHEET
GEPHI OVERVIEW PRINTED NETWORK
BIG SCREEN
RUN IT
notebook, terminal
label
label
image url label web entitites image domain
label
label
*size
*color
[attributes]
[in-degree, degree,out-degree]
label
label
annotation
Title
color 1
color 2
Design by Beatrice Gobbo I Concept by Janna Joceli Omena](https://image.slidesharecdn.com/caisclosingpresentation-makingmethodspdf-220211144150/95/Making-methods-with-vision-APIs-online-data-network-building-lessons-learnt-10-638.jpg)
Making methods with vision APIs, online data & network building (lessons learnt)
- 1. Making methods with vision APIs, online data & network building (lessons learnt ) Dr Janna Joceli Omena 26 January 2022 I CAIS fellowship closing presentation I Computer Vision Networks
- 2. 2 Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks PERSONAL • Living in Bochu m J.J.Omena@fcsh.unl.pt https://thesocialplatforms.wordpress.com / RESEARCH INTERESTS Digital method s Software studies Visual network analysi s Methodological innovation EDUCATION Universidade Nova de Lisboa I UT Austin Portugal Progra m PhD thesis in Digital Media studies: Digital methods and technicity-of-the-mediums. PROFESSIONAL Invited associate professor in digital media and methods, NOVA University Lisbo n iNOVA Media Lab & Public Data La b
- 3. 3 Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks 1. Situating the computer vision network approach to study image collections
- 4. Developing digital visual methods for social and medium research Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks computer vision networks as an ensemble of computational mediums, data, methods, research, and technical practices orchestrated by the researcher(s) . a computer vision network approach offers three different forms of interpreting the same image collection : 1. the content of image itself & its web cultural -social-political contexts 2. the site of image audiencing & to whom the images matte r 3. the site of image circulation
- 5. 5
- 6. 6 Computer Vision Networks Developing digital visual methods for social and medium research Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks Aims Objectives Results 1. Interrogate the potentials and limitations of computer vision APIs for social and medium research • Organise data sprints with field experts to try and test the computer vision network approach, while mapping its potentials/limitations. • Develop research software to facilitate the processes of network building with vision APIs outputs. (in collaboration w/ Jason Chao) • Create a method recipe to explain the approach, testing and trying it in different contexts Research software to invoke multiple vision APIs and query image collections in collaboration with Jason Chao 2. Develop accessible and reproducible visual methodologies with digital methods
- 7. 7 Chao, T. H. J. (2021). Memespector GUI: Graphical User Interface Client for Computer Vision APIs (Version 0.2) [Software]. Available from https://github.com/jason-chao/ memespector-gui. Memespector GUI
- 8. Offline Image Query and Extraction Tool 8 Chao, T. H. J. & Omena, J. J. (2021). Of fl ine Image Query and Extraction Tool (Version 0.1) [Software]. Available from https://github.com/jason-chao/of fl ine- image-query.
- 9. 9 Computer Vision Networks Developing digital visual methods for social and medium research Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks Aims Objectives Results 1. Interrogate the potentials and limitations of computer vision APIs for social and medium research • Organise data sprints with field experts to try and test the computer vision network approach, while mapping its potentials/limitations. • Develop research software to facilitate the processes of network building with vision APIs outputs. (in collaboration w/ Jason Chao) • Create a method recipe to explain the approach, testing and trying it in different contexts Research software to invoke multiple vision APIs and query image collections in collaboration with Jason Chao A method recipe to build and interpret computer vision networks (first step to propose a conceptual-methodological model) 2. Develop accessible and reproducible visual methodologies with digital methods
- 10. 10 The method recipe The method protocol analyse images QUERY OR QUERIES DESIGN RESEARCH QUESTION DATASET DESIGN PROCESS VISUALISATION PROCESS VISUAL NETWORK ANALYSIS images URL, engagement metrics, timestamps .csv EXTRACT IMAGE METADATA IMAGES METADATA alike Image Tagnet Explorer, Tumbrl Tool, Google Image Extractor according to analysed the digital platform DIGITAL PLATFORM Alike Instragam, Tumbrl, Google Image ] IMAGES URL .csv, .tsv, .txt file with URLs ORGANISE AND CLEAN IMAGE METADATA alike Excel, Google Spreadsheet DOWNLOAD IMAGES FROM URL DownThemAll or similar RESIZE IMAGES BulkResize or similar] FOLDER OF IMAGES [.native format] FOLDER OF RESIZED IMAGES [.native format] alike Google Vision API USE THE WEB VISION API SERVICE CREATE AN API KEY BUILD OR USE AN ALREADY EXSISTING SCRIPT script (.py or .php) es: “from google.cloud import vision (...) NETWORK [.gexf] .csv VISION API METADATA image url label web entitites image domain BUILD THE EDGES/NODES TABLE BUILD THE NETWORK Table2Net or manually] VISUALISE THE NETWORK Gephi] Gephi image preview plugin] *Image Network Plotter Script works as well on Pyhton] INSTALL THE GEPHI IMAGE PREVIEW PLUGIN Gephi + domain knowledge] [es: ForceAtlas2 Spatialisation] VISUALISE AND SPATIALIZE IT ROUGHLY CHANGE NETWORK APPEREANCE ACCORDING TO DATA ADD PICTURES TO THE NETWORK label VISUALLY ANALYSE THE NETWORK PRESENT/STAGE THE NETWORK label GO BACK TO THE RESEARCH QUESTIONS label label *size *color [attributes] [in-degree, degree,out-degree] EXPORT THE NETWORK IN AN VECTOR FORMAT EDIT ADD ANNOTATIONS ADD A TITLE ADD KEY REFINE COLORS ADJUST LABEL SIZE [.svg, .eps] Adobe Illustrator, Inkscape] mediums / software technical practice output researcher intervention VISUALISATION PRESENTATION label label annotation Title color 1 color 2 GEPHI DATA LAB SPREADSHEET GEPHI OVERVIEW PRINTED NETWORK BIG SCREEN RUN IT notebook, terminal label label image url label web entitites image domain label label *size *color [attributes] [in-degree, degree,out-degree] label label annotation Title color 1 color 2 Design by Beatrice Gobbo I Concept by Janna Joceli Omena
- 11. 11 Prerequisites I method recipe 📣No coding skills are demanded • be willing to practice new methods • get familiar with a range of software • bring your own computers to the classroom • work with spreadsheets and a list of research software
- 12. 12 Computer Vision Networks Developing digital visual methods for social and medium research Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks Aims Objectives Results 1. Interrogate the potentials and limitations of computer vision APIs for social and medium research • Organise data sprints with field experts to try and test the computer vision network approach, while mapping its potentials/limitations. • Develop research software to facilitate the processes of network building with vision APIs outputs. (in collaboration w/ Jason Chao) • Create a method recipe to explain the approach, testing and trying it in different contexts Research software to invoke multiple vision APIs and query image collections in collaboration with Jason Chao A method recipe to build and interpret computer vision networks (first step to propose a conceptual-methodological model) 2. Develop accessible and reproducible visual methodologies with digital methods Peer-review article + data sprint reports + tutorials Digital Methods Initiative (Summer/Winter Schools)
- 13. 13 Trying-and-testing the method recipe I Digital Methods Summer School 2021 I Projects Teaching methods and software-using I Digital Methods Summer School 2021 I Tutorials
- 14. 14 Diseña // No. 19 (2021): Visual Methods for Online Images: Collection, Circulation, and Machine Co-Creation Methodological proposal I Article: The potentials of Google Vision API-based Networks to Study Natively Digital Images
- 15. 15 Computer Vision Networks Developing digital visual methods for social and medium research Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks Aims Objectives Results 1. Interrogate the potentials and limitations of computer vision APIs for social and medium research • Organise data sprints with field experts to try and test the computer vision network approach, while mapping its potentials/limitations. • Develop research software to facilitate the processes of network building with vision APIs outputs. (in collaboration w/ Jason Chao) • Create a method recipe to explain the approach, testing and trying it in different contexts Research software to invoke multiple vision APIs and query image collections in collaboration with Jason Chao A method recipe to build and interpret computer vision networks (first step to propose a conceptual-methodological model) 2. Develop accessible and reproducible visual methodologies with digital methods Peer-review article + data sprint reports + tutorials Digital Methods Initiative (Summer/Winter Schools) Innovative network building and reading techniques
- 16. Network building without image s nodes as computer vision outputs and _vision apis service s _web environments where images come from (e.g. social media, meme generator platforms) _platform data (e.g. location based-data pointing to countries, images posted by public social media accounts) 16 Reading network visualisation through fixed layers of interpretatio n _centre: what the actors have in common _periphery: the unique characteristics of the actors _mid-zone: specific aspects shared among actors or the shadow of a particular acto r
- 17. 17 Network building nodes as web entities and meme environment s
- 18. 18 Reading network visualisatio n Meme project I DMI Winter School 202 2 I Method recipe, network building and reading techniques: Janna Joceli Omena I Network visualisation: Marco Valli
- 19. 19 Reading network visualisatio n Meme project I DMI Winter School 202 2
- 20. 20 Reading network visualisatio n Meme project I DMI Winter School 202 2
- 21. 21 Reading network visualisatio n Meme project I DMI Winter School 202 2
- 22. 22 This technique enables image cross-platform analysis Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
- 26. 26 Mapping Deepfakes project I DMI Summer School 202 1 Technique conceptualisation: Janna Joceli Omen a Network building and analysis: Giulia Tucci nodes as countries and web entities associated with the image dataset (Tweets using #deepfakes and imgs)
- 27. 27 nodes as populists political leaders (Instagram accounts) and web entities associated with their recent publications (images) Political leaders on Instagram I Project by TISE master students Course: Introduction to Digital Method s Technique conceptualisation and recipe: Janna Joceli Omena Network building and analysis: Ilya Lavrov, Chiara Miozzo, Marie Palaffre, Franziska Schranz
- 28. 28 Meme project I DMI Winter School 2022 I Method recipe, network building and reading techniques: Janna Joceli Omena
- 29. Developing digital visual methods for social and medium research Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks Aims Objectives Results 1. Interrogate the potentials and limitations of computer vision APIs for social and medium research • Organise data sprints with field experts to try and test the computer vision network approach, while mapping its potentials/limitations. • Develop research software to facilitate the processes of network building with vision APIs outputs. (in collaboration w/ Jason Chao) • Create a method recipe to explain the approach, testing and trying it in different contexts Research software to invoke multiple vision APIs and query image collections A method recipe to build and interpret computer vision networks (first step to propose a conceptual-methodological model) 2. Develop accessible and reproducible visual methodologies with digital methods Peer-review article & data sprint reports Tutorials Digital Methods Initiative (Summer/Winter Schools Innovative network building and reading techniques
- 30. 30 Three lessons from making, testing & validatin g the computer vision network approach Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
- 31. 31 1 a technicity perspective is not optional but a crucial task Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
- 32. 32 2 to mechanise methodolog y does not exclude human decisions, intervention and engagement (on the contrary!) Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
- 33. 33 3 what is still unknown about vision APIs functioning or its limitations are not methodological bias bu t aspects to be taken into account through method implementation Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
- 34. 34 What’s next?
- 35. 35 Warren Pearce Carlo De Gaetano José Moreno Rita Sepúlveda Jason Chao Amazing group of Erasmus Mundus Master Students I The seminal class of the course introduction to digital methods at NOVA University Lisbon CAIS team & fellows Thank you =) Beatrice Gobbo I Johannes Breuer I Richard Rogers Giulia Tucci I Francisco Kerche
- 36. CAIS visiting NOVA University Lisbon