HANA SPS07 Text Analysis
Download as PPTX, PDF1 like1,750 views
The document summarizes new and improved features in SAP HANA SPS 07 related to text analysis. Key updates include the ability to create custom dictionaries and configurations, improved indexing throughput through greater scalability, and expanded language support for features like social media extraction, numerical extraction, and voice of customer analysis.
HANA SPS07 Text Analysis
- 1. What´s New? SAP HANA SPS 07 Text Analysis (Delta from SPS 06 to SPS 07) SAP HANA Product Management November, 2013
- 2. Agenda New or Improved Text Analysis Features Custom dictionaries Custom configurations Indexing throughput Improved Language Coverage Social Media extraction for Japanese & Simplified Chinese Numerical extraction for Simplified Chinese Core extraction for Russian Voice of Customer for Simplified Chinese Related Topics Fulltext search Fuzzy search © 2013 SAP AG. All rights reserved. Public 2
- 3. New or Improved Text Analysis Features
- 4. New Custom Dictionary Support You can now specify your own entity types and names to be used with text analysis, which may be critical for particular industries or data domains Single custom dictionary may support all languages or a single language Custom dictionaries reside in the HANA repository and benefit from its life cycle management Steps 1. 2. 3. 4. 5. Choose the project to contain the new dictionary in the Development perspective of SAP HANA Studio. Enter or select a parent folder and enter the dictionary file name in the Wizard. Your text analysis dictionary file is created locally and opens as an empty file in the text editor. Enter your text analysis dictionary specification into the new file and save it locally. Commit your new dictionary. The dictionary is now synchronized to the repository as a design time object and the icon shows the dictionary is committed. Activate once you have finished editing your dictionary. The dictionary is created in the repository as a runtime object and the icon shows the dictionary is activated. This allows you and others to use the dictionary. If you haven’t done so previously, you will need to create a custom text analysis configuration as well… © 2013 SAP AG. All rights reserved. Public 4
- 5. New Custom Configuration Support You can now customize the features and options used for text analysis rather than using the predefined configurations: LINGANALYSIS_BASIC LINGANALYSIS_STEMS LINGANALYSIS_FULL EXTRACTION_CORE EXTRACTION_CORE_VOICEOFCUSTOMER Custom configurations allow you to suppress the default output and incorporate custom dictionaries. You can either: Create a new XML configuration file within SAP HANA Studio Copy one of the predefined configurations and modify it © 2013 SAP AG. All rights reserved. Public 5
- 6. Greater Indexing Throughput Improved scalability of the highlighted preprocessing steps: File filtering – converting binary document formats to text/HTML Tokenization – decompose word sequence, e.g. “the quick brown fox” -> “the” “quick” “brown” “fox” Stemming – reduction of tokens to linguistic base form, e.g. houses -> house; ran -> run Linguistic analysis 30% less time Depending upon hardware configuration – part-of-speech identification, e.g. quick: Adjective; houses: Plural Noun Utilizes more threads and efficient data transfers Applies to all text analysis configurations 50% greater throughput Depending upon hardware configuration © 2013 SAP AG. All rights reserved. Public 6
- 8. Available Text Analysis Configuration Options Language LINGANALYSIS_FULL EXTRACTION_CORE EXTRACTION_CORE_VOICEOFCUSTOMER Arabic LINGANALYSIS_BASIC LINGANALYSIS_STEMS X Catalan X X Chinese (Simplified) IMPROVED IMPROVED Chinese (Traditional) X X Croatian X X Czech X X Danish X X Dutch X English Farsi X French German Greek X X X Hebrew X X X Hungarian X X X Italian X Japanese IMPROVED X Korean X Norwegian (Bokmal) X X Norwegian (Nynorsk) X X Polish X X X Portuguese X Romanian X X X Russian IMPROVED X Serbian X X Slovak X X Slovenian X X Spanish Swedish X X Thai X X X Turkish X X X © 2013 SAP AG. All rights reserved. Public 8
- 9. Improved Social Media Extraction for Japanese & Simplified Chinese Identifies with high recall and precision SOCIAL_MEDIA entities with corresponding offsets Tags SOCIAL_MEDIA entities such as IDs (@MyTwitterName) or topics (#MyWeiboKeyword) Distinguishes between SOCIAL_MEDIA entities and emoticons like @__@ Distinguishes between SOCIAL_MEDIA entities and emails like myname@domain.com Respects important Weibo and Twitter differences, Ex: #W-TOPIC# vs. #T-TOPIC1 #T-TOPIC2 © 2013 SAP AG. All rights reserved. Public 9
- 10. Improved Numerical Extraction for Simplified Chinese Better identifies numerical entities with special characters CURRENCY – expressions denoting amounts of money – 33.8万元 – 港币五千万 – 一百四十四亿七千万美元 DATE – minimally composed of a number and month name – 7月2日 – 十月十七日 MEASURE – expressions – 二百五十六公斤 – 5.5米 TIME – clock times and time expressions – 8时 – 3点零5分 © 2013 SAP AG. All rights reserved. Public 10
- 11. Additional Predefined Core Extractions for Russian TITLE PERSON PEOPLE LANGUAGE President Barak Obama Greeks Greek ADDRESS1 ADDRESS2 LOCALITY REGION@MINOR REGION@MAJOR COUNTRY CONTINENT GEO_FEATURE GEO_AREA 245 First Street Floor 16 Cambridge, MA 02142 Cambridge Napa Country Connecticut Brazil South America Mount Fuji Scandinavia ORGANIZATION@COMMERCIAL ORGANIZATION@EDUCATIONAL ORGANIZATION@OTHER PRODUCT TICKER AT&T University of Washington FBI iPhone NYSE:SAP SOCIAL_MEDIA@TWITTER_ID SOCIAL_MEDIA@TWITTER_TOPIC DATE DAY MONTH YEAR TIME TIME_PERIOD HOLIDAY 2/14/2011 Monday June 2011 3:47pm 3 days, from 9 to 5pm Memorial Day CURRENCY 17 euros MEASURE PERCENT 217 meters 4% PHONE URI@EMAIL URI@IP URI@URL 617-677-2030 john.smith@sap.com 165.14.2.0 http://sap.com Syntactic Entities: NOUN_GROUP PROP_MISC big umbrella Cup o’ Soup @SAP #HANA © 2013 SAP AG. All rights reserved. Public 11
- 12. Improved Voice of Customer Extraction for Simplified Chinese The following major fact types are classified: Sentiments: expression of a customer’s feelings about something Problems: a statement about something which impedes a customer’s work Requests: expression of a customer’s desire for an enhancement/change Profanity: defines a set of pejorative vocabulary Emoticons: expression of someone's feelings about the whole sentence or situation Focuses on finer extraction of online reviews and implementing customer feedback Dramatic overall improvement in stances and topics Recall and precision testing results jumped significantly higher © 2013 SAP AG. All rights reserved. Public 12
- 13. Disclaimer This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP’s strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent. © 2013 SAP AG. All rights reserved. Public 13
- 14. Thank you Contact information Anthony Waite SAP HANA Product Management AskSAPHANA@sap.com To get the best overview of what’s new in SAP HANA SPS 07, read this blog.
- 15. © 2013 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Please see http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark for additional trademark information and notices. © 2013 SAP AG. All rights reserved. Public 15
- 16. © 2013 SAP AG. Alle Rechte vorbehalten. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Einige der von der SAP AG und ihren Distributoren vermarkteten Softwareprodukte enthalten proprietäre Softwarekomponenten anderer Softwareanbieter. Produkte können länderspezifische Unterschiede aufweisen. Die vorliegenden Unterlagen werden von der SAP AG und ihren Konzernunternehmen („SAP-Konzern“) bereitgestellt und dienen ausschließlich zu Informationszwecken. Der SAP-Konzern übernimmt keinerlei Haftung oder Gewährleistung für Fehler oder Unvollständigkeiten in dieser Publikation. Der SAP-Konzern steht lediglich für Produkte und Dienstleistungen nach der Maßgabe ein, die in der Vereinbarung über die jeweiligen Produkte und Dienstleistungen ausdrücklich geregelt ist. Keine der hierin enthaltenen Informationen ist als zusätzliche Garantie zu interpretieren. SAP und andere in diesem Dokument erwähnte Produkte und Dienstleistungen von SAP sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und verschiedenen anderen Ländern weltweit. Weitere Hinweise und Informationen zum Markenrecht finden Sie unter http://www.sap.com/corporateen/legal/copyright/index.epx#trademark. © 2013 SAP AG. All rights reserved. Public 16