Trend Detection and Visualization and Custom Search Applications
Download as PPTX, PDF3 likes1,825 views
The document discusses trend detection and visualization, focusing on statistical methods for numerical data and automated systems for textual data. It also explores custom search applications like Apache Solr and semantic search technologies. Various techniques for visualizing trends, such as line graphs and themerivers, are highlighted alongside approaches for identifying emerging trends in literature and web resources.
Trend Detection and Visualization and Custom Search Applications
- 1. Trend Detection and Visualization and Custom Search Applications Seminar for PG PUSHPIN Pranav Kadam (6641525) Universität Paderborn January 12, 2012
- 2. Overview • Trend Detection Trend Detection in Numbers Trend Detection in Text Trend Visualization • Custom Search Applications Apache Solr Semantic Search Linked Data Approach Trend Detection and Visualization and Custom Search Applications 2
- 3. Overview • Prototypes • Q&A Trend Detection and Visualization and Custom Search Applications 3
- 4. Trend Detection Trend Detection and Visualization and Custom Search Applications 4
- 5. Trend Detection What is a trend? • A general direction in which something is changing • An inclination • A pattern of gradual change in a condition over time • A trend is always associated with time often described using ‘time series‘ • Long term change in the mean level of a ‘time series‘. Trend Detection and Visualization and Custom Search Applications 5
- 6. Trend Detection Trend Analysis • Practice of collecting information and trying to detect trend in it • Process of identifying pattern in behavior of a time series by minimising noise • Useful in forecasting future events • Science of studying changes in social patterns E.g. Google Trends, Youtube Trends, trendwatching.com, Facebook Insights, Tag Cloud(on PG PUSHPIN blog) Trend Detection and Visualization and Custom Search Applications 6
- 7. Trend Detection Trend Detection in Numbers Trend Detection and Visualization and Custom Search Applications 7
- 8. Trend Detection in Numbers Time series and statistical methods • Time series: ordered sequence of values at equally spaced time intervals • Trend detection in numbers: Statistical methods to interpret time series and determine behavior • Assumption: pattern in past data can be used to forecast future data points • Models: AutoRegressive(AR), Integrated(I), Moving Average(MA) Trend Detection and Visualization and Custom Search Applications 8
- 9. Trend Detection in Numbers Moving Average • Average of time series data taken at consecutive periods • New data in, old data out as the series progresses E.g. MA of temperature for six months: Temp from January to June, February to July, March to August, and so on. • Minimizes temporal fluctuations • Establishes trend, distinguishes any value above or below trendline • Applications in fields of Financial analysis, Trade, Economics, Mathematics Trend Detection and Visualization and Custom Search Applications 9
- 10. Trend Detection in Numbers Moving Average • Simple Moving Average: Plain average of data points over specific no. of periods • Period selected can be short, medium or long according to interest (E.g. standard periods of SMA for stock market analysis is 50 days or 200 days) • Longer the period gives smoother curve but increases the lag • SMA always lags behind the latest data point Trend Detection and Visualization and Custom Search Applications 10
- 11. Trend Detection in Numbers Moving Average • Exponential Moving Average: Weight applied to the data pointa to reduce the lag • Weight decreases exponentially and never reaches zero • EMA has less lag and is more sensitive to the changes in data points • SMA vs EMA: Though difference is apparent, either one cannot be stated as better over the other MA preference depends on objectives & time horizon Trend Detection and Visualization and Custom Search Applications 11
- 12. Trend Detection Trend Detection in Text Trend Detection and Visualization and Custom Search Applications 12
- 13. Trend Detection in Text Trend detection system • Emerging Trend: Topic area growing in interest and utility over time • Study of emerging trend dependent on automated process • TD system processes collection of textual data and identifies upward(growing), downward(falling) or sideway(constant) tendency • TD then highlights the emerging topics in trial period Trend Detection and Visualization and Custom Search Applications 13
- 14. Trend Detection in Text Trend detection system • Trend detection methods can be classified as: Fully-automatic Semi-automatic • Fully-automatic systems: It generates a list of emerging topics from the input(collection of texual data) Reviewer examines data & evidence provided to conclude actual emerging trends Results supported with graphical visualization Trend Detection and Visualization and Custom Search Applications 14
- 15. Trend Detection in Text Trend detection system • Semi-automatic: User inputs a topic System outputs the evidence that helps to determine that the topic is emerging or not Evidence provided either as a summary or a descriptive report Trend Detection and Visualization and Custom Search Applications 15
- 16. Trend Detection in Text Useful models, schemes and tools • Term-Document Matrix • Scheme: Term Frequency – Inverse Document Frequency (tf-idf) • Latent Semantic Analysis • Science Citation Index or Web of Science database • Inspec, Compendex database Trend Detection and Visualization and Custom Search Applications 16
- 17. Trend Detection in Text Approches for Trend Detection 1. Tracing a trend via citation linkages: Determine a potential trend or select a topic of interest Find recent documents on the topic Examine whether they really discuss the topic Extract keywords Fetch abstract of the documents those are frequently referenced using citation information Examine abstract to verify relation with topic Trend Detection and Visualization and Custom Search Applications 17
- 18. Trend Detection in Text Approches for Trend Detection 1. Tracing a trend via citation linkages: Examine the references used above and make a subset where author names are referenced in more than, say, 3 documents As an improvement, query the repositories of citation linkage information and other sources Graph document frequency, repeated authors and no. of venues by year Trend Detection and Visualization and Custom Search Applications 18
- 19. Trend Detection in Text Approches for Trend Detection 1. Tracing a trend via citation linkages: Years with overall higher document frequency are likely to have points where trend is emerging Finally, to determine trend, apply a series of thresholds like atleast one repeated author, atleast 10 venues present, etc. Trend Detection and Visualization and Custom Search Applications 19
- 20. Trend Detection in Text Approches for Trend Detection 2. Using web resources: Select a main topic area first Knowledge in this area is essential to identify trends in later stages Validate it as a possible research area using sources like Inspec database Search workshop websites and technical papers for discussions on the main topic area Trend Detection and Visualization and Custom Search Applications 20
- 21. Trend Detection in Text Approches for Trend Detection 2. Using web resources: Search web using helper terms like most recent contribution, hot topic, cutting edge strategy, etc Again search an indexing database with main topic ‘AND‘ newly found candiate trend from year of origin to current year Trend Detection and Visualization and Custom Search Applications 21
- 22. Trend Detection in Text Approches for Trend Detection 2. Using web resources: If document frequency increases over the years, the candidate trend is a genuine trend x If documents from same author appear in different years its not a trend Trend Detection and Visualization and Custom Search Applications 22
- 23. Trend Detection Trend Visualization Trend Detection and Visualization and Custom Search Applications 23
- 24. Trend Visualization Trend visualization techniques • Trends can be visualized using Line graphs Bar graphs Word clouds Frequency tables Sparklines Histograms Trend Detection and Visualization and Custom Search Applications 24
- 25. Trend Visualization Other ways to visualize trends • ThemeRiver Visualizes thematic variations over time Changing widths depict changes in thematic strength of the associated documents Flow represents time Colors represent themes Vertical section represents an ordered time slice Trend Detection and Visualization and Custom Search Applications 25
- 26. Trend Visualization Other ways to visualize trends • ThemeRiver Trend Detection and Visualization and Custom Search Applications 26
- 27. Trend Visualization Other ways to visualize trends • ThemeRiver Assigning same color group to related themes simplify its tracking Trend Detection and Visualization and Custom Search Applications 27
- 28. Trend Visualization Other ways to visualize trends • SparkClouds SparkClouds= Sparklines + Tag Clouds Sparkline, characterized by small size and high data density, visualize trends and variations in a simple condensed way Trend Detection and Visualization and Custom Search Applications 28
- 29. Trend Visualization Other ways to visualize trends • SparkClouds Tag clouds are text based visualizations showing frequency, popularity or importance of words Trend Detection and Visualization and Custom Search Applications 29
- 30. Trend Visualization Other ways to visualize trends • SparkClouds Sparklines are added to tag clouds to represent trend across series of tag clouds Overview of trends provided in limited space Its compact and aesthetic Trend Detection and Visualization and Custom Search Applications 30
- 31. Custom Search Applications Trend Detection and Visualization and Custom Search Applications 31
- 32. Custom Search Application Apache Solr • Open source search platform from Apache Lucene project • Provides full text search, faceted search, dynamic clustering, database integration, rich document handling, geo-spatial search • High scalability, distributed search • The core of search and navigation engine of some of the world‘s largest internet sites Trend Detection and Visualization and Custom Search Applications 32
- 33. Custom Search Application Apache Solr • Written in Java, runs as a standalone search server within a servlet container like Jetty or Tomcat • REST-like API eases its use with any prog. language • Input: XML, JSON or binary over HTTP(GET) • Output: XML, JSON or binary • Highly customizable Trend Detection and Visualization and Custom Search Applications 33
- 34. Custom Search Application Apache Solr • Operations: Indexing data Updating data Deleting data Querying data Sorting Higlighting Faceted search Trend Detection and Visualization and Custom Search Applications 34
- 35. Custom Search Application Semantic Web • An extension to current Web • Information is given well-defined meaning • Goes beyond media objects to link people, places, events, organizations, etc. • Resources connected by multiple relations • Data modeled using directed labeled graph • Based on W3C‘s RDF, it does quering and exchanging instance data in RDF using SOAP Trend Detection and Visualization and Custom Search Applications 35
- 36. Custom Search Application Semantic Web 9°C temp located in type USA City San Francisco Apple Inc. birth place Steve Jobs type Company Businessman died on Pixar February 24, 1955 October 5, 2011 Trend Detection and Visualization and Custom Search Applications 36
- 37. Custom Search Application Semantic Search • Context-based search results • Can possibly enhance, but cannot replace the traditional navigational search • Disambiguation • Data divided as ontological data and instance data • Determines meaning of every word and establishing a context between them to achieve coherence for a sentence Trend Detection and Visualization and Custom Search Applications 37
- 38. Custom Search Application Semantic Search • Search Methodologies: RDF Path Traversal Keyword Concept Mapping Graph Patterns Logics Fuzzy Concepts, Fuzzy Relations, Fuzzy Logics • Examples Hakia, SenseBot, DeepDyve Trend Detection and Visualization and Custom Search Applications 38
- 39. Custom Search Application Linked Data Approach • Linked data: method of publishing structured data that can be interlinked • Based on HTTP and URIs, extended to be read by computers • Components: URIs HTTP RDF Serialization formats (RDFa, RDF/XML, N3) Trend Detection and Visualization and Custom Search Applications 39
- 40. Custom Search Application Linked Data Approach • KiWi – a Linked Media Framework • Easy to setup server application bundling Semantic Web technologies • Consists of LMF core and LMF modules Trend Detection and Visualization and Custom Search Applications 40
- 41. Custom Search Application Linked Data Approach • KiWi LMF core: Use URIs as names for things. Use HTTP URIs, so that people can look up those names. When someone looks up a URI, provide useful information, using the standards (RDF, SPARQL). Include links to other URIs, so that they can discover more things. Trend Detection and Visualization and Custom Search Applications 41
- 42. Custom Search Application Linked Data Approach • KiWi LMF module: LMF Semantic Search(highly configurable Semantic Search service based on Apache SOLR) LMF Linked Data Cache (implements a cache to the Linked Data Cloud) LMF Reasoner (implements a rule-based reasoner that allows to process Datalog-style rules over RDF triples) Trend Detection and Visualization and Custom Search Applications 42
- 43. Prototypes Trend Detection and Visualization and Custom Search Applications 43
- 44. Questions and Answers Trend Detection and Visualization and Custom Search Applications 44
- 45. Thank you! Trend Detection and Visualization and Custom Search Applications 45
Editor's Notes
- #43: Datalog is a query and rule language for deductive databases that syntactically is a subset of Prolog