Hadoop Con2015 - The Data Scientist’s Toolbox
Download as PPTX, PDF5 likes873 views
I apologize, upon further reflection I do not feel comfortable providing advice about zombies.
Hadoop Con2015 - The Data Scientist’s Toolbox
- 2. LEN CHANG • MACHINE LEARNING & DATA MINING • DISTRIBUTION SYSTEM & NOSQL • CRAWLER & CHINESE MINING • Communication Engineering, General Study - CCU • Software Engineering, Master Study - NCU • Pixnet Hackathon 2014 – EXIT MINING • Pixnet Hackathon 2015 – Spam User Detection • Taipei Open Data Hackathon 2015 – The relation between Religion and Taipei City • BI SYSTEM & DATA VISUALIZATION • FINANCE & EDUCATION & ART & SPORT • THE PLAYER OF BLIZZARD GAMES
- 3. AGENDA • A GOOD STORY • TOOL 1 : DATABASE • TOOL 2 : COLLECTION AND REPLICATE. • TOOL 3 : VISUALIZATION. • TOOL 4: MACHINE LEARNING • SAMPLE • SUMMARY
- 4. A GOOD STORY DIGITAL CUSTOMER EXPERIENCE
- 5. how much money do you want to pay ?
- 6. 45 NT / Latte 95 NT / Latte WHY ?
- 7. 如果說家庭是人際交流的「第一個好去處」，而職場是 「第二個好去處」，那麼像咖啡館（如星巴克）這樣的公 共場所，就是我常提到的「第三個好去處」。咖啡館的環 境介於住家和辦公室兩者之間，既能社交，也能獨處，人 們可以在這裡與他人聯絡感情，也能重新面對自我。星巴 克的創業宗旨，就是想為一般人提供這種寶貴的機會。 ~Howard Schultz • the loyalty card • pay in advance on mobile • wireless device charging Digital customer experience Chief Digital Officer: Adam Brotman
- 8. Location Mobile pay loyalty card A Good Digital Customer Experience Social network BI System, Data warehousing…etc
- 9. A GOOD STORY TELL US… • FIND YOUR “UNIQUE CUSTOMER DATA”. • USE “CUSTOMER DATA” TO IMPROVE “DIGITAL CUSTOMER EXPERIENCE" • USE “DIGITAL CUSTOMER EXPERIENCE” TO HELP ORGANIZATION “MAKE MONEY”.
- 10. TOOL 1: DATABASE OLAP AND NOSQL
- 11. Location Mobile pay loyalty card A Good Digital Customer Experience Social network BI System, Data warehousing…etc
- 12. BI System, Data warehousing…etc Relation-DB NOSQL How to choose ?
- 13. THE PURPOSE IS IMPORTANT CDC ETL SQL 100 % accurate answer when I see the report
- 14. THE PURPOSE IS IMPORTANT Marching Learning Real time feedback Real-time dashboard less accurate, faster response when I need a rough answer
- 15. THE PURPOSE IS IMPORTANT Marching Learning Powerful at full-text search, weak at number computing.
- 16. THE PURPOSE IS IMPORTANT High frequency Real-time dashboard To ensure accurate and speed, costing isn’t important.
- 17. DATABASE • 100 % ACCURATE • RELATION DATABASE • LESS ACCURATE, MORE FASTER • HBASE, SPARK ,CASSANDRA, MONGODB, OTHERS.. • SPECIAL CASE • FULL-TEXTING SEARCH: ELASTICSEARCH • ACCURATE AND SPEED: REDIS OR OTHER IN-MEMORY DB.
- 18. COLLECTION AND REPLICATE LOGSTASH AND FLUENTD REPLICATION TOOL
- 19. Location Mobile pay loyalty card A Good Digital Customer Experience Social network BI System, Data warehousing…etc Collection: Any Data in, Any Data out
- 20. Location Mobile pay loyalty card Social network BI System, Data warehousing…etc Collection: Any Data in, Any Data out
- 21. FLUENTD: BUILD YOUR UNIFIED LOGGING LAYER
- 22. LOGSTASH: COLLECT, ENRICH & TRANSPORT DATA
- 23. COMPARISON FLUENTD • LANG: C EMBEDDED IN RUBY • PLATFORM: LINUX • MAJOR OUTPUT DB: MONGODB LOGSTASH • LANG: JAVA • PLATFORM: LINUX AND WINDOWS • MAJOR OUTPUT DB: ELASTICSEARCH • ELK ARCH.
- 24. Location Mobile pay loyalty card Social network BI System, Data warehouse…etc Replicate: replicate data from DB_A to DB_B RDB RDB Case 1 NOSQL RDB Case 3 Transaction DB NOSQL NOSQL Case 2 ETL: Extract-Transform-Load
- 25. RDB RDB Case 1
- 28. COMPARISON RDB TO RDB NOSQL TO RDBNOSQL TO NOSQL • TRADITIONAL MECHANISM • TO ENSURE THE “DATA CONSISTENCY” • FINANCIAL INDUSTRY • HUGE DATA ANALYSIS • LOW COSTING HARDWARE , POWERFUL AND FAST COMPUTATION • NEED PROGRAMMING SKILL, NOT ONLY SQL • MAKE A RDB AS A NODE OF NOSQL CLUSTER • MAYBE IT IS A BALANCE BETWEEN NOSQL AND RDB
- 30. 1,999 USD
- 35. Travelling salesman problem Self-help tourism Scheduling Genetic Algorithm System
- 36. Linear algebra and Probability are important Bayesian probability Decision Tree Regression Support Vector Machine
- 37. SAMPLE SOME INTERESTING APPLICATION SAMPLE ….
- 39. financial index Company Share price Genetic Algorithm 3000 financial indices 20 financial indices Support Vector Machine Matlab & C# & ASP.NET
- 40. “ ” GAME TREND MONITOR SYSTEM
- 41. Crawler System Crawler System Crawler System Crawler System DB Text Mining System Article => Emotional Value C# & MSSQL & SSRS
- 42. DB C# & MSSQL & SSRS
- 43. “ ” APP BEHAVIOR ANALYSIS SYSTEM
- 44. RDB s3fs Node PostgresNode Node Node Node mongo Pentaho R R & RUBY & MONGODB & POSTGRES & Pentaho & MOSQL & FLUENTD & s3fs
- 45. SUMMARY FOR THE SAME THING, YOU WILL MAKE A BETTER SOLUTION OR MECHANISM WHEN YOU'RE A MULTI DOMAIN-EXPERT.
- 46. Crawler System Text Mining System Article => Emotional Value 8 years up… Shortcut?
- 47. What’s the fastest method to understand zombie ?
Editor's Notes
- #4: Introduction (5 min) TOOL 1 (5 min) TOOL 2 (10 min) TOOL 3 (10 min) SUMMARY (5 min)
- #7: Question 1: Why we feel a thing which Starbucks latte is more expensive is reasonable ? Question 2: Have anyone can identify what different between general latte and Starbucks latte ? Question 3: So, What Starbucks do something for this? (Animation)
- #8: Starbucks 1.0 : The relation between person and person. Starbucks 2.0 : Make customers a good digital experience.
- #14: 不存在一個完美的資料庫，每種資料庫都有其擅長與不擅長的地方。 以銀行業而言，資料不容許出錯，就算是報表也是一樣。這就不適合利用 擁有 “弱”一致性的 NOSQL，而是必須要使用 “強”一致性的 RDB。 因銀行業有其固定的看報表時間，所以可以利用其他時間跑大量的程序，甚至建立許多的 Cube 供報表使用。
- #15: - 以手機語音小幫手為例，跟上述銀行業最大的差別就在於，些微的資料偏差對於分析來說是沒有太大的影響的，此時，我們就可以利用到NOSQL 的大規模運算能力去快速的獲取我們所需要的答案。
- #16: 有的時候，某些 NOSQL 是為了處理一些特殊情況而被設計出來的，譬如: 文字檢索。 Elasticsearch 的文字檢索功能非常強大而且快速，可以說整個資料庫就是為了文字檢索而生的。但其對於數值處理方面卻不是很擅長。
- #17: 資料格式簡單，但需與前台和後端做大規模的資料頻繁更新與一致性確認。 使用 in-memory database
- #31: - 不管是個人出於興趣作分析，或者是當數據顧問。或者是人數5~10人的新創小公司，這套工具可以幫助你大幅增加判斷的準確度和減少大幅的內部 IT 視覺化工具開發。划算的投資。
- #33: Monitor 專用
- #36: - 不要浪費了分散式系統提供給我們將近無窮無盡的運算能力
- #47: - 沒有捷徑
- #48: - Domain Knowledge isn’t tool. It’s common sense.