Garewal Harnessing the Power of a Semantic Index at JSTOR
Download as PPTX, PDF1 like1,002 views
The document discusses various initiatives by Ithaka, a not-for-profit organization, to leverage digital technologies in the academic community, particularly through JSTOR and its services like Portico and Artstor. It outlines the functionalities of the JSTOR thesaurus and the MAISTRO software, which includes automated and manual indexing, as well as the text analyzer tool for discovering related content. Additionally, it describes the use of Latent Dirichlet Allocation (LDA) for topic modeling to enhance document discoverability through inferred topics and named entity recognition.
Garewal Harnessing the Power of a Semantic Index at JSTOR
- 1. May 9, 2018 Sharon Garewal, Taxonomy Manager (Sharon.Garewal@ithaka.org) Harnessing the power of a semantic index at JSTOR
- 2. ITHAKA is a not-for-profit organization that helps the academic community use digital technologies to preserve the scholarly record and to advance research and teaching in sustainable ways. JSTOR is a not-for-profit digital library of academic journals, books, and primary sources. Ithaka S+R is a not-for-profit research and consulting service that helps academic, cultural, and publishing communities thrive in the digital environment. Portico is a not-for-profit preservation service for digital publications, including electronic journals, books, and historical collections. Artstor provides 2+ million high-quality images and digital asset management software to enhance scholarship and teaching.
- 3. Presentation Outline JSTOR Thesaurus MAIstro Rulebase JSTOR Labs Text Analyzer LDA topic models
- 4. Topics appear on search results page. Topics appear at the document level (i.e. journal article, book chapter, etc.) Topics are linked to a topic page at the document level. The topic page provides a description/image (if available and correct) from Wikipedia. Topic cards highlight the top four occurring thesaurus terms within the document. Thesaurus on JSTOR Platform
- 8. User feedback Ability for users to provide feedback on the topics. Can choose individual topics as inaccurate or thumbs up/thumbs down all topics. Report is exported each week and terms flagged as inaccurate are reviewed. 7
- 9. MAIstro MAIstro is the thesaurus software developed by Access Innovations. MAI=Machine Aided Indexing. There are three components of the software: • Thesaurus master-This is where terms and the hierarchy are maintained. • Rule Builder-Automated and manual entry of rules for terms which allows for customization. • Test MAI-Area to test documents against the thesaurus and the rule base for suggested terms.
- 12. Indexing and display Content requirements Terms will not be applied to foreign language documents. Terms will not be applied to documents that are coded as miscellaneous or book reviews. Some documents will not have terms applied since the document is a shorter length. Term must appear 3x (default) before it is triggers a match. Suggested terms are given for each document; default of 10 terms are given. • Ordered by number of times the term appears within the document.
- 13. JSTOR Labs works with partner publishers, libraries and labs to create tools for researchers, teachers and students that are immediately useful – and a little bit magical.
- 14. Text Analyzer - beta
- 15. Text Analyzer Analyzes arbitrary text to find related content in JSTOR archive Drag-n-drop File select
- 16. Text Analyzer • Text Analyzer extracts topics and named entities from submitted text to find related/similar document in the JSTOR archive • Topics are based on the terms in the JSTOR Thesaurus
- 17. Text Analyzer Text is submitted via: • Direct input • Copy/paste • Local file • Drag and drop from local computer filesystem or a web URL • Photo of text, via phone camera A variety of document types are supported: • PDF • MS-Word • HTML • RTF • Plain text, Powerpoint, and Excel • Images (on-the-fly OCR is performed)
- 18. Another example My bookshelf at work Topics inferred Using a smartphone photo as input…
- 19. Image analysis My bookshelf at work Topics inferred
- 20. Text Analyzer recommendations Recommendations are based on a ”best fit” of all prioritized terms (topics and entities) and weights • The selection of documents in results represents an ‘OR’ of documents containing one or more terms • Results ordering is based on a score representing the number of terms matched, the importance of the term(s) to the document (based on LDA weight) and the user-specified importance • A user is able to quickly refine the terms and weights to tailor the results to a specific need Values used in relevancy and ranking calculations are available for inspection
- 21. Latent dirichlet allocation (LDA) • LDA is one of the most common algorithms for topic modeling The Latent part of LDA comes into play because in statistics, a variable we have to infer rather than directly observing is called a "latent variable". We're only directly observing the words and not the topics, so the topics themselves are latent variables (along with the distributions themselves).* • LDA is based on the concept that: • Every document is a mixture of topics • Every topic is a mixture of words • LDA is a mathematical method for estimating both of these at the same time: • Finding the mixture of words that is associated with each topic, • while also determining the mixture of topics that describes each document * https://www.quora.com/What-is-a-good-explanation-of-Latent-Dirichlet-Allocation
- 22. Latent dirichlet allocation (LDA) • Model training • Can be supervised or unsupervised • When performed unsupervised a predefined number of topics are identified and are represented by word probabilities • Supervised training involves the use of a tagged corpus where the tags will be used as topic labels • For topics we use a subset of the JSTOR Thesaurus • For the model training documents we’re now using Wikipedia articles associated with each topic • Topic inferencing • Using a trained topic model, ‘latent’ topics in a document can be inferred using the words in the text • Topics are expressed as probability distribution
- 23. What is an LDA topic? It’s simple…
- 24. OK, the math isn’t so simple but conceptually a topic is just a set of “word” relationships climate temperature earth ice warming global weather atmosphere climate_change ocean cycle oscillation carbon greenhouse model scientist water age ice_age atmospheric event tropical gas pacific heat dioxide carbon_dioxide pattern average region wave surface extreme variation air classification cold precipitation latitude cooling global_warming land radiation solar science greenhouse_gas rainfall determine condition enso report emission current dry theory variability force summer polar infrared annual future north range mass climatic feedback atlantic northern record sea rise scientific natural evidence scale factor planet winter cold_wave glacial air_mass warm climate_model regional cfc extreme_weather climatology niño interglacial oceanic assessment pollution phase absorption location published america ancient result energy arrhenius surface_temperature climate_oscillation vegetation seasonal trend moon south shift activity sun assessment_report climate_pattern humid hurricane fluctuation anomaly methane conclude decadal maritime tree arctic concentration month short infrared_radiation glacier future_climate monsoon forecasting global_temperature continental milankovitch orbital water_vapor vapor james estimate normal observe maximum variable pdo heat_wave pacific_ocean climatologist tree_ring arid convince forcings holocene ice_sheet cloud fourier climate_sensitivity icehouse weather_event wmo southern_oscillation climate_climate solar_variation climate_cycle north_america croll human_emission icehouse_climate agassiz enso_event climate_index global_climate climate_variability mid_latitude paleoclimatology thornthwaite köppen current_climate indian_ocean niño_southern_oscillation niño_southern interglacial_period climate_classification electromagnetic_radiation term_climate ocean_atmosphere wind_shear fossil_fuel ramanathan wetherald manabe keeling absorbing_infrared james_croll charpentier scientific_opinion buckland level_pressure sea_level_pressure inter_decadal tropical_pacific decadal_oscillation mjo climate_science ozone_depletion nao extreme_weather_event change_climate climate_proxy east_pacific annual_basis ice_cap subarctic_climate oceanic_climate humid_subtropical modern_climate climate_zone bergeron polar_ice regular_cycle scientific_literature lake_bed current_interglacial temperature_fluctuation warm_period shorter_term classification_include climate_force milankovitch_cycle projected_increase excessive_heat heatwave bioclimatology cfc_focused dioxide_molecule carbon_dioxide_molecule absorbing_infrared_radiation lovelock_speculate james_lovelock_speculate scientist_james_lovelock_speculate scientist_james_lovelock british_scientist_james_lovelock british_scientist_james core_drilled ice_core_drilled particulate_pollution aerosol_pollution sea_core deep_sea_core david_keeling charles_david_keeling charles_david callendar varves högbom infrared_absorption measure_infrared cycle_lasting venetz perraudin change_climate_change climate_change_climate_change climate_change_climate change_science climate_change_science century_scientist background_climate bake_crust british_scientist langley james_lovelock cfc_molecule chlorofluorocarbon_cfc tyndall john_tyndall extreme_event scientist_james hothouse energy_budget teleconnections sst_anomaly For example, the top “words” associated with the topic Climatology
- 25. Named Entity Recognition (NER) • Entities in a submitted text are identified and available for document selection • Persons • Locations • Organizations • Results from multiple entity recognition engines are merged during analysis • IBM Alchemy • OpenCalais (Thompson Reuters) • OpenNLP (Apache) • Stanford NER
- 28. Future opportunities • Multilanguage topic inferencing (being tested in Text Analyzer) • Expanding the LDA topic model training set • Integration of LDA Topic modeling and MAIstro Rulebase • Named entities on all documents 27
- 29. Thank You