Applications of Semantic Technology in the Real World Today
Download as PPT, PDF0 likes703 views
The document discusses the applications of semantic technology to address common business challenges related to data, highlighting both structured and unstructured data technologies. It emphasizes the need for semantic technologies to discover relationships, process data contextually, and support informed decision-making in complex situations, such as compliance and risk management in financial services. Real-world case studies illustrate the successful implementation of semantic solutions for issues like disambiguation of data and enhanced profiling across disparate information sources.
Applications of Semantic Technology in the Real World Today
- 1. Applications of Semantic Technology in the Real World Today Amit Sheth, CTO, Semagix Inc
- 2. Common Business Challenges Inability to see the “Big Picture” Unidentified threats Missing opportunities Optimal value of information unrealized Diminished productivity & effectiveness Unrevealed information assets “ Informed” decision-making impossible Reactive decision-making Damage control
- 3. Problem: Structured Data Technologies These technologies are… Well-defined Predictable Quantifiable Data-driven Time/resource intensive Enterprise Technologies based on What these technologies are NOT capable of… Handling loosely coupled information Managing unplanned information sources Harmonizing many small pockets of corporate data Handling the other 80% of unstructured data WHAT YOU DON’T KNOW, BUT NEED TO KNOW What these technologies ARE capable of… Mission critical applications Often transaction based Large corpus of similar information Handling well-planned, repeatable data applications WHAT YOU KNOW OLAP OLPT Inventory Data Mining Relational Spreadsheets Data Warehouse
- 4. Problem: Unstructured Data Technologies These technologies are… Highly Restricted Superficial Uninsightful Not actionable Time / resource intensive Enterprise Technologies based on What these technologies are NOT capable of… Discovering hidden relationships of different types and at multiple levels between entities Contextually processing structured & unstructured data Establishing powerful links between disparate documents Delivering unambiguous proactive insight to make informed decisions WHAT YOU DON’T KNOW, BUT NEED TO KNOW What these technologies ARE capable of… Keyword-based processing of mainly uninstructed content Detect co-occurrence of entities in documents Attempt to only provide back information that you asked for Tell you what else from that set is similar to what you asked for WHAT YOU KNOW Classification Search Clustering Entity Extraction Fact Extraction Summarization Vocabularies
- 5. Things to Consider About the Semantic (Web) Technologies Build Ontology Build Schema (model level representation Populate with Knowledgebase (people, location, organizations, events) Automatic Semantic Annotation (Extract Semantic Metadata) Any type of document, multiple sources of documents Metadata can be stored with or sparely from documents Applications: search ( ranked list of documents of interest (semantic search), integrate/portal, summarize/explain , analyze, make decisions Reasoning techniques: graph analysis, inferencing Types of content/documents Use of standards Scalability Performance opscenter
- 6. Ontology-driven Information System Lifecycle Schema Creation Ontology Population Metadata Extraction BSBQ Application Creation Analytic Application Creation Ontology API MB KB Building a scalable and high performance system with support for: Ontology creation and maintenance Ontology-driven Semantic Metadata Extraction/Annotation Utilizing semantic metadata and ontology Semantic search/querying/browsing Information and application integration - normalization Analysis/Mining/Discovery – relationships Semantic Technology Solves These Challenges
- 7. Upper ontologies: modeling of time, space, process, etc Broad-based or general purpose ontology/nomenclatures: Cyc, CIRCA ontology (Applied Semantics), SWETO , WordNet ; Domain-specific or Industry specific ontologies News: politics, sports, business, entertainment Financial Market Terrorism PharmaO GlycO (Glycomics); PropeO (Proteomics) GO (nomenclature), NCI (schema), UMLS (knowledgebase), … Application Specific and Task specific ontologies Anti-money laundering, NeedToKnow, (Employee or Vendor Whetting) Equity Research Repertoire Management Fundamentally different approaches in developing ontologies: schema vs populated; community efforts vs reusing knowledge sources Types of Ontologies (or things close to ontology)
- 8. More sophisticated semantic technologies exploit ontologies and Provide scalability and flexibility Handle all types of data (unstructured, semi-structured, structured) Create SmartData – enhancing raw data with context and relationships Accommodate SmartQuerying – flexible, intelligent querying Enable powerful enterprise decision making Evolution of Meta Data
- 9. Real-World Applications (case studies)
- 10. Global Bank Aim Legislation (PATRIOT ACT) requires banks to identify ‘who’ they are doing business with Problem Volume of internal and external data needed to be accessed C omplex name matching and disambiguation criteria Requirement to ‘risk score’ certain attributes of this data Approach Creation of a ‘risk ontology’ populated from trusted sources (OFAC etc); Sophisticated entity disambiguation Semantic querying, Rules specification & processing Solution Rapid and accurate KYC checks Risk scoring of relationships allowing for prioritisation of results Full visibility of sources and trustworthiness
- 11. Ahmed Yaseer appears on Watchlist member of organization works for Company Ahmed Yaseer: Appears on Watchlist ‘FBI’ Works for Company ‘WorldCom’ Member of organization ‘Hamas’ The Process Watch list Organization Company Hamas WorldCom FBI Watchlist
- 12. Global Investment Bank Example of Fraud Prevention application used in financial services World Wide Web content Public Records BLOGS, RSS Un-structure text, Semi-structured Data Watch Lists Law Enforcement Regulators Semi-structured Government Data User will be able to navigate the ontology using a number of different interfaces Scores the entity based on the content and entity relationships Establishing New Account
- 13. Law Enforcement Agency Aim Provision of an overarching intelligence system that provides a unified view of people and related information Problem Need to create unique entities from across multiple disparate, non-standardised databases; Requirement to disambiguate ‘dirty’ data Need to extract insight from unstructured text Approach Multiple database extractors to disambiguate data and form relevant relationships Modelling of behaviours/patterns within very large ontology (6Mn+ entities) Solution Merged and linked case data from multiple sources using effective identification, disambiguation, and link analysis Dynamic annotation of documents Single query across multiple datasets 360 view of an individual and relevant associations
- 14. Application of bespoke and pre-configured ‘profiles’ for detailed investigation Profile Creation Complex Querying Summary of Results Investigation Profile Creation Complex Querying Summary of Results Investigation Complex querying and characteristic modelling across information sources
- 15. User configurable scoring profiles Profile based on direct matching with case characteristics Profiling based on link analysis through indirect relationships with other cases and information Profile Creation Complex Querying Summary of Results Investigation
- 16. Free text searching across aggregated information sources Gisondi, white ford expedition, main street, assault, traffic offences Profile Creation Complex Querying Summary of Results Investigation
- 17. Unified view of direct and indirect results that best match the complex query and the profile Profile Creation Complex Querying Summary of Results Investigation
- 18. Knowledge Annotation of known entities from within free text Direct and indirect relationship scoring driven by risk weightings Aggregated knowledge from disparate sources Profile Creation Complex Querying Summary of Results Investigation
- 19. Scoring of key characteristics to drive relevance to original profile and query Identification of investigation path Visualisation of results Profile Creation Complex Querying Summary of Results Investigation
- 20. 3D navigation of relationships and knowledge around a query
- 21. Example of a base ontology
- 22. Key Characteristics of the Key Cases Scalable end-to-end platform driven by domain specific ontologies Expressive representation with named relationships Populated ontologies with millions of instances Sophisticated entity disambiguation of knowledge extracted from multiple knowledge sources Self-maintaining ontologies – updated as needed
- 23. Key Characteristics of the Key Cases Unified 360-degree view of entities across heterogeneous information sources Domain specific semantic metadata extraction and enhanced annotation of heterogeneous documents and heterogeneous content Semantic linking from internal and 3 rd party content/sites Full visibility of sources and trustworthiness Comprehensive & high performance analytical processing Relationship linking of information Custom scoring of relationships within information
- 24. Help create/maintain populated ontologies that capture domain terms (do these map to classification level?) Automatic annotation of data; possibly value added metadata enhancement (could be consistent with any metadata standard) Providing insight into the documents (show annotated data, link concepts in documents with ontologies or context for search) Show conceptual similarity between documents Rule-based or pattern-based processing Discovering links
- 25. QUESTIONS?
- 26. Semagix Product Architecture RAW DATA XML Thin Agile Applications Model Integrate Enhance Deliver SMART Services SMART Works Freedom SMART Services Smart Data Ontology SMART Central SMART Search SMART Explore SMART Connect SMART Notify SMART View
- 27. Technical Capabilities Unified Ontology Representation Language Expressiveness Ontology Quality and Freshness Populated Ontology Size Data: Type and Amount Metadata Extraction: type Computation: query expressiveness (over metadata and ontology), rules, ranking Visualization
Editor's Notes
- #7: Semantic Web in a Nutshell: - Ontology as the centerpiece - Metadata that associate meaning to content - Computing (complex querying, inferencing, other reasoning) that support semantic applications
- #8: CENTRAL ROLE OF ONTOLOGIES Ontology represents agreement, represents common terminology/nomenclature Ontology is populated with extensive domain knowledge or known facts/assertions Key enabler of semantic metadata extraction from all forms of content: unstructured text (and 150 file formats) semi-structured (HTML, XML) and structured data Ontology is in turn the center price that enables resolution of semantic heterogeneity semantic integration semantically correlating/associating objects and documents
- #9: Large scale metadata extraction and semantic annotation is possible. IBM WebFountain [Dill et al 2003] demonstrates the ability to annotate on a Web scale (i.e., over 2.5 billion pages), while Semagix Freedom related technology [Hammond et al 2002] demonstrates capabilities that work for a few million documents per day per server. However, the general trade-off of depth versus scale applies. Storage and manipulation of metadata for millions to hundreds of millions of content items requires database techniques with the challenge of improving performance and scale in presence of more complex structures
- #13: (a) Serve global population of 500 users (B) Complete all source checks in 20 seconds or less © Integrate with enterprise single sign-on systems (d) Meet complex name matching and disambiguation criteria (e) Adhere to complex security requirements Results: Rapid, accurate KYC checks; Automatic audit trails; Reduction in in false positives; Streamlines and enhances due diligence of potential high risk accounts
- #14: Requirements: (a) Merge and link case data from multiple sources to a taxonomy using effective identification, disambiguation, and analysis; (b) Ability to use pre-defined/investigation-specific case studies for search and match © Positive and negative searching of cases (d) Ability to explore case data starting from any entity via link analysis Results: Superior, faster identification of prolific offenders; Better prioritization of cases; Greater investigator productivity and effectiveness