conceptClassifier For SharePoint Driving Business Value
- 1. Leveraging Metadata to Improve Findability, Records Management, and Compliance in SharePoint Martin Garland, Presidentmarting@conceptsearching.com
- 3. Challenges in Knowledge Intensive OrganizationsMetadata Drives
- 4. Search
- 5. Compliance & Records Management
- 6. Privacy and Security Exposures
- 8. Manual Metadata Approach will fail
- 9. The ROI of Managing Unstructured Content Assets
- 10. Benefits of metadata consistency, automated classification and taxonomy management
- 11. Technology Suite
- 12. Roadmap to SharePoint 2010
- 13. Case Study – US Air Force Medical ServiceConcept Searching, Inc.Company founded in 2002
- 14. Product launched in 2003
- 15. Focus on management of structured and unstructured information
- 16. Privately held and profitable – no funding
- 17. Growth rate of 35% in 2008 and in excess of 100% for 2009
- 18. Founders and management team with company since inception
- 19. Technology
- 20. Automatic concept identification, content tagging, auto-classification, taxonomy management
- 21. Only statistical vendor that can extract conceptual metadata
- 22. 2009 and 2010 ‘100 Companies that Matter in KM’ (KM World Magazine)
- 23. KMWorld ‘Trend Setting Product’ of 2009
- 24. Locations: US, UK, & South Africa
- 25. Client base: Fortune 500/1000 organizations
- 26. Managed Partner under Microsoft global ISV Program - “go to partner” for Microsoft for auto-classification and taxonomy management
- 27. Microsoft Enterprise Search ISV , FAST Partner
- 28. Product Suite: conceptSearch, conceptTaxonomyManager, conceptClassifierChallenges in Knowledge Intensive CompaniesMetadata Generation
- 30. Inability to harvest and reuse intellectual capital
- 31. Lack of organizational memory – most organization’s don’t know what they already know
- 32. Semantic identification of concepts
- 34. Inability to share expertise and knowledge across geographic and cultural barriers due to inconsistent nomenclature
- 35. Inability to deliver consistency across global boundaries and even within practice areas
- 36. Folksonomy dilutes metadata value
- 37. Search alone does not deliver ‘findability’
- 38. Inability to manage knowledge assets so that relevant information can be found and used
- 39. Retrieval of information based on keywords or proximity does not deliver relevance as to what concepts the document contains
- 40. Continuously replacing your search engine will not fix search
- 41. Inability to control and manage content to improve back end processes such as records managementThe Dilemma Between Precision and Recall Two key performance measures for information retrieval
- 42. Most information retrieval technologies are less than 22% accurate for both precision and recall
- 43. Ideal is to have them balanced
- 44. Recall is the retrieval of all items that are relevant to the query
- 45. Precision is the retrieval of only those items that are relevant to the query
- 46. Higher precision leads to missing items that may be relevant but use a different vocabulary
- 47. Higher recall leads to the retrieval of too many items that may be unrelated to the query Automatic Concept Identification has the ability to increase precision with no loss of recall
- 48. Metadata Consistency Drives Business Agility Findability & Enterprise Content ManagementFindability every time and at a lower cost per click
- 49. Deliver a robust content management approach maximizing SharePoint technologies
- 50. Guided navigation, related topics based on conceptsIdentification of Assets for Data Security, eDiscovery, and Litigation PreparednessReduced litigation for ediscovery
- 51. Costs associated with data breachesRecords Management & Compliance Eliminate inconsistent meta-tagging
- 53. Lower costs for managing records and informationCollaboration and Enterprise 2.0Increase collaboration and productivity through integration of diverse repositories
- 54. Improve information sharing and expert identification reducing rework and recreation
- 55. Integration with external repositories, web sites delivering a single searchinterface
- 56. Metadata Drives Actionable SearchKeyword search captures only 33% of relevant information. Consistent, meaningful metadata ensures all relevant information related to key words will be returned.
- 57. Users can’t navigate to information. Taxonomies provide consistent guided navigation for end users to extract relevant information even in external content. Taxonomy navigation is 36%-48% faster and more efficient than lists.
- 58. Vocabulary normalization across diverse geographies and cultures causes issues and inhibits sharing of knowledge and expertise due to nomenclature.
- 59. Case Study: Fortune 500 firm realized that search alone did not solve findability issues. Implemented conceptClassifier to secure and manage content in a policy-compliant manner, eliminated end user tagging, delivered the ability to rapidly build and deploy taxonomies, and to normalize vocabulary across global boundaries.KNOWLEDGE WORKERS CHALLENGES~ 15% of their time is spent duplicating information.~ 25% of their time is spent searching.~ 40% can not easily find the information they require to do their job.The cost to a 500 employee company is$2.4 million per year in inefficiencies and lost productivity. Gartner Group
- 60. Metadata Ensures Compliance & Records ManagementProtects the organization by eliminating end user adoption issues
- 61. Ensures adoption to any enterprise regulation for external agencies or where compliance is mandatory
- 62. Easy Integration with Microsoft Records Center. Ensures the long term usefulness of the records and enforcement of life cycle management.
- 63. Automatic assignment of Records Retention codes
- 64. Optional updating of Content Type based on the metadata contained within the documents
- 65. Case Study: US Air Force Medical Service eliminated all manual metadata tagging and uses conceptClassifier to automatically generate semantic metadata, assign record retention codes based on the metadata within the content, automatically change the Content Type and migrate documents to the RMCOMPLIANCE & RECORDS MANAGEMENT CHALLENGES~ The average cost of manually tagging one item is estimated at $4.00 - $7.04~IDC estimates that only 50% of content is correctly meta-tagged ~ It costs and organization $180 per document to recreate it when it is not tagged correctly and cannot be found~ Poor information quality costs organizations 10% to 20% of operating revenues
- 66. Metadata Helps Avoid Data Privacy & Security Issues conceptClassifier for SharePoint ensures compliance by automatically identifying Personally Identifiable Information (PII), Protected Health Information (PHI) or any metadata that is considered by the organization to be confidential.
- 67. Migrates content to a secure location where Windows Rights Management Services is applied to the file in the new location
- 68. Optionally can change the Content Type during the classification process
- 69. The taxonomy standardizes the process of identifying all possible privacy data exposures at the time of content creation and modification (digital and handwritten).
- 70. Case Study: conceptClassifier for SharePoint extracted 2,000+ documents with sensitive information, from a redacted sample pilot data set. By human error, these 2,000 records contained real social security numbers and real employee information. These documents were identified in the proof of concept to the client in front of executive management.DATA BREACHES & EXPOSURES CHALLENGES~ Average cost of a data breach is $6.3 million and ranges from $225K to $35 million.~ Average cost per exposed record is $197 and ranges from $90-$305 per record.~ 70% of breaches were due to a mistake or malicious intent by an organization’s own staff.~ Healthcare provider - $7 million, TJX Companies - $256 million, ValueClick - $2.9 million.
- 71. Metadata Drives CollaborationAbility to add structure to chaos
- 72. Generate weighted results from diverse repositories such as human resources records, time and billing, project documentation, content authorship, team structures, user profiles
- 73. Results are generated based on the most experienced and knowledgeable individual for that specific topic or skill set aggregated from diverse repositories
- 74. Single logical view for expertise search across diverse information stores
- 75. Removes geographical boundaries through vocabulary normalization
- 76. Case Study: Professional Services firm with over 39K employees across 36 countries uses conceptClassifier for expert identification to identify and utilize in-house consultants for projects as opposed to outsourcing – increasing utilization of staff by 5% to 10%Collaboration and Enterprise 2.0~ Nearly 80% of executives believe collaboration is important but needs to be managed~Email storage costs $500GB per year – a Fortune 100 manufacturing company saved $2.6 million per year by implementing collaboration solutions~ Up to 90% of content from premium paid publication database services is available for free on-line~ Only 25% of executives describe their organization as effective at sharing knowledge across boundaries
- 77. A Manual Metadata Approach Will Fail 95% Of The Time
- 78. An Automated Metadata Approach Drives Business ValueCreate enterprise metadata framework/model
- 79. Average return on investment minimum of 38% and runs as high as 600% (IDC)
- 80. Apply consistent meaningful metadata to enterprise content
- 81. Incorrect meta tags costs an organization $2,500 per user per year – in addition potential costs for non-compliance (IDC)
- 82. Guide users to relevant content with taxonomy navigation
- 83. Savings of $8,965 per year per user based on an $80K salary (Chen & Dumais)
- 84. Use automatic conceptual metadata generation to improve Records Management
- 85. Eliminate inconsistent end user tagging at $4-$7 per record (Hoovers)
- 86. Improve compliance processes, eliminate potential privacy exposuresConcept Based Metadata GenerationCompound Term Processing – the ability to extract ‘concepts in context’
- 87. Only statistical metadata generation and classification company that can extract concepts from content as it is created or ingestedtriple heart bypassconceptClassifier will generate conceptualmetadata by extracting multi-word terms that identifies ‘triple heart bypass’ as a concept as opposed to single keywords
- 88. Search will return results based on the concept even if the exact terms are not contained in the document (i.e. ‘coronary artery surgery’, ‘heart surgery’)
- 89. Metadata can be used by any search engine index or any application/process that uses metadataTripleBaseballThreeHeartOrganCenterBypassHighwayAvoid
- 90. conceptClassifier for SharePointconceptClassifier for SharePoint
- 91. Automatic identification, extraction, and tagging of content with concepts
- 92. Intelligent auto-classification based on concepts not keywords
- 93. Enterprise class Taxonomy Management, uniquely based on concept identification
- 94. Integration
- 95. A technology platform that runs natively in SharePoint 2007 and 2010
- 96. Microsoft Enterprise Search, FAST ESP
- 97. Microsoft Office
- 98. Windows Server 2008 R2 FCI
- 99. Component – Automatic Conceptual Metadata Generation
- 100. Automatically generates and extracts metadata including keywords, acronyms, and multi-word terms that form concepts and convey meaning
- 101. Component – Automated Classification
- 102. Rules based categorization module
- 103. Real-time classification of individual pieces of content aligned to defined business structure(s)
- 104. Automatically classifies documents to multiple nodes in multiple taxonomies
- 105. Highly scalable, fast real-time classification
- 106. Classifier may be called via web services, or by other related applications (e.g.: FAST pipeline stage)
- 107. Based upon identified and extracted concepts proven to be more effective than keyword classifiersconceptClassifier for SharePointComponent – Taxonomy Management
- 108. Hierarchical taxonomy structure with ontological features and ability to import standard structures such as OWL and RDF
- 109. Rapid taxonomy creation and maintenance
- 110. Automatic concept identification extracted from clients own content to populate taxonomy(s)
- 111. Class clues generated from client’s unique document corpus
- 113. Maintenance via clue suggestion, integrated search, and instant feedback
- 114. Class weighting influenced by parent, child, sibling
- 115. AJAX based user interface
- 116. Taxonomy structure and classification results held in SQL database
- 118. Based on organizationally defined metadata provides the ability toautomatically change the SharePoint Content Type based on the presenceof the metadata within documents (if a document contains Protected Health Information (PHI) it will automatically change the Content Type to ‘PHI’)Technology
- 119. SOA compliant and delivered as Web Parts
- 120. API is based entirely on Web Services and all information is exchanged in XML T
- 121. Taxonomy formats are based on Web Ontology Language (OWL). Since the server is stateless it also works with all failover and load balancing hardware and software.
- 122. Roadmap from SharePoint 2007 to 2010There is no auto classification of metadata (i.e. no way to auto apply term set values) it is a manual process
- 123. There is no way to automatically generate metadata when it is created or ingested
- 124. Same problem with end users adding inconsistent metadata
- 125. Taxonomy management is a manual process
- 126. Taxonomy maintenance requires significant resources to maintain and change as business changes
- 128. Properties (current flat lists) become hierarchical “Term Sets” –
- 129. Term Sets provide capability for faceted search and hierarchical navigation: Regions Country/State, Business Unit/Departments, Band Names/Album Names, TV Show Titles/Characters
- 130. conceptClassifier fully supports SharePoint 2010 EMM as the primary location for taxonomy definitions with no need to Import/Export
- 131. Changes to the taxonomy structure using Microsoft tools will be immediately visible in conceptClassifier and vice versaconceptClassifier for FAST SearchDelivers all functionality in conceptClassifier for SharePoint including:Improves faceted search results as facets are based on concepts aligned within the taxonomy
- 132. Provides taxonomy browse capabilities based on the nodes within the corporate taxonomy(s)
- 133. Provides accurate metadata filters such as numeric range searching and wildcard alphanumeric matching
- 134. Improves search outcomes by placing conceptual metadata in the FAST Search index to increase relevancy of search results
- 135. Enables import of FAST Entities into the conceptClassifier taxonomy manager to fine-tune them with metadata generated from your own content and nomenclature
- 136. Runs natively as a FAST Pipeline Stage eliminating integration and customization issues
- 137. Eliminates vocabulary normalization issues across global boundaries through controlled vocabulariesCase Study – US Air Force Medical ServiceUS Air Force Medical Service
- 138. Initially deployed conceptClassifier to power Knowledge Portal with over 65K users
- 139. Controlled vocabulary consists of over 27K unique keywords, metadata, and multi-word fragments generated by conceptClassifier
- 140. Have expanded use of technology to provide the following:
- 141. Automatically organize data assets prior to migration to SharePoint - identifies duplicates, PII, PHI before migration
- 142. Automatically extract metadata and classify to the various taxonomies
- 143. Provide Microsoft Enterprise Search and FAST indexes the conceptual metadata to improve findability
- 144. Identification and location of sensitive information (PII, PHI, Confidential) and migrates content to new location where Windows Rights Management Services are applied
- 145. Automatically tags and classifies content based on semantics contained within the actual document of record and optionally updates the Content Type
- 146. Eliminates end user tagging issues
- 147. Records are declared in a consistent manner
- 148. Only real records are migrated to the RM system
- 149. Only real records are managed for disposition
- 150. All other content stays in the collaborative portal
- 151. Freedom of Information Act (FOIA) compliance
- 152. eDiscoveryUSAF Human Performance ClearinghouseGOAL : Leverage Existing USAF, AFDW, and AFMS License Agreements to Enable IM, RM, & Privacy & Security ComplianceRequirementsDoDD 8320 (Data Sharing in a Net-Centric DoD)
- 153. DoDD 5015 (Records Management)
- 154. USAF Privacy Act Program & HIPAA
- 155. Freedom of Information Act (FOIA)Distribution Statement A: Approved for public release; distribution is unlimited. 311 ABG/PA No. 09-488, 16 Oct 2009
- 156. Screen Shots
- 157. Navigation – Auto Complete, Taxonomy & FacetedMicrosoft Enterprise Search/FAST ESP can utilize highly relevant compound term metadata
- 158. Browsable taxonomy navigation via Concept Searching Web Part
- 159. Faceted navigation (integrated with Microsoft CodePlex)FAST ESP SearchCross taxonomy navigation filter
- 160. Taxonomy Browse
- 161. Conceptual metadata supplied to FAST search indexAutomatic Classification & Metadata TaggingContent is automatically tagged with semantic metadata and uploaded to SharePoint
- 162. Content is automatically classified to one or more nodes in one or more taxonomies
- 163. Documents are automatically classified to multiple categories
- 164. Editable from within SharePoint & the Concept Searching Taxonomy ManagerFull Support for Content TypesEliminates time consuming manual metadata definition
- 165. Enforces governance, policies, and drives workflows in line with business processes
- 166. Enables different taxonomies to be assigned to different Content Types
- 167. Authorized users have complete control over automatically generated metadataAutomatic Update of Content Types/WorkflowEvent HandlerBased on a pre-defined Event Handler, the Content Type can be automatically changed when classified.When organizationally defined metadata is identified within content the Content Type Updater will automatically change the Content TypeOffice IntegrationFully integrated with Microsoft Office & Exchange
- 168. Content automatically tagged with semantic metadata stored in custom properties
- 169. Content automatically classified to corporate or departmental taxonomies
- 170. Delivers governance at the desktop, improves ECM
- 171. Automatic metadata generation or optionally authorized users can change the classificationAggregates Multiple Content SourcesAbility to specify multiple file sources including:
- 172. SharePoint
- 173. Web Sites
- 175. File stores
- 176. Can also include RSS feeds
- 177. Automatically classify and place semantic metadata in search engine indexTaxonomy & Concept Based Metadata GenerationConceptual metadata automatically generated from the organization’s own content and used as clues to build out the taxonomyHierarchical view of content
- 178. Content will be automatically classified to one or more nodes based on concepts within the content
- 179. Reduces time to develop, build, and maintain a taxonomy by as much as 80%
- 180. Can import industry standard taxonomiesAutomatic Clue SuggestionManual entry of class clues is available
- 181. Suggest Clues for Class
- 182. Search the document corpus and identify documents that are about the new node to be included or excluded
- 183. Clues can be single words, multi-word terms (concepts) or acronyms Classification and Automatic Clue FeedbackClicking on a clue will display a document summary and the extract where the clue occurs in the context of the document.
- 184. The end user can either keep the clue, remove it, or modify the weightingLeveraging Metadata to Improve Findability, Records Management, and Compliance in SharePoint Martin Garland, Presidentmarting@conceptsearching.com