SlideShare a Scribd company logo

Suffix Array@Solr勉強会

N
nobu_k

1. A suffix array is an array containing all suffixes of a string sorted in lexicographic order. It allows for efficient string searches, including prefix searches, substring searches, and near neighbor searches. 2. The suffix array can be used to find the starting positions of all occurrences of a pattern or substring within the text in time linear to the pattern length. It supports finding the longest common prefix between two strings in constant time. 3. Suffix arrays are useful for text indexing and string searching, including applications like full text search, bioinformatics, data compression, string mining, and more. They allow locating patterns in strings and calculating similarities between strings.

1 of 26
Downloaded 22 times
1
2
Most read
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Suffix Array Solr 2011/12/19 1
• (@nobu_k) • Preferred Infrastructure (PFI FI) • • • Sedue(2 ) 2
Suffix Array • Suffix Array(SA): • ( ) 1 • Sedue • SA • • +Sedue • ” - ” 3
• • • ( ) • n-gram(q-gram) • 4
Suffix Array • • • n-gram • • 5
Suffix( ) 0: mississippi 1: ississippi 2: ssissippi 3: sissippi mississippi 4: issippi 5: ssippi 6: sippi 7: ippi 8: ppi 9: pi 10: i 6
Suffix Array 0: mississippi 10: i 1: ississippi 7: ippi 2: ssissippi 4: issippi 3: sissippi 1: ississippi 4: issippi 0: mississippi 5: ssippi 9: pi 6: sippi 8: ppi 7: ippi 6: sippi 8: ppi 3: sissippi 9: pi 5: ssippi 10: i 2: ssissippi 7
10: i 7: ippi • mississippi ’si’ 4: issippi 1: ississippi • ’si’ 0: mississippi 9: pi • 8: ppi 6: 3: sippi sissippi • 5: ssippi 2: ssissippi • 3 6 8
10: i SA[i]: 7: ippi 4: issippi 10 7 4 1 0 9 8 6 3 5 2 1: ississippi T[i]: 0: mississippi 9: pi m i s s i s s i p p i 8: ppi 6: sippi 3: sissippi 6 5: ssippi T[SA[6]] 2: ssissippi → T[8] → “ppi” 9
(1/3) T[i]: 1 2 3 ... n SA SA[i] 10
(2/3) RedBull !! 1. RedBull *2 RedBull SA[i] 2. RedBull 1 2 3 ... n 11
(3/3) 3. RedBull 1 2 3 ... n 4. ( 1, 3), ( 2, 4), ( 3, 2),...,( n, 2) 12
• SA • + • /n-gram • SA • 13
SA • (n-gram ) • • n-gram • • • “THIS IS IT” • proximity 14
SA • • • • • HDD • ( ) • • 15
• • ( ) • SAIS • • HDD • (dc3, dc7) • Sedue Haskell C++ • @tanakh++ 16
• ( ) • • • 1 100GB/day • Sedue • SA n-gram • n-gram • SA n-gram • 17
HDD • HDD • OK • • • SSD • SSD • Sedue 20 (80MB) • SA[i] 18
VS 1. SA • 2. • SSD+ 500 3. • O(N) CPU 4. • • malloc 19
• Sedue 1 56 • : 40 • : 16 (UTF-16) • 2 3 • • = • • SSD • 20
SA • • 4(+1) • 2-gram • • % OK • • ” ” • 21
• • • • • 22
: groonga • Sedue groonga • • • Sedue groonga!! 23
: • • • (http://jubat.us/) • http://github.com/jubatus • @JubatusOfficial • with NTT PF 24
: Fluentd • Ruby • Treasure Data, Inc. • @frsyuki, @kzk_mover • Solr • gem install fluentd • Visit http://fluentd.org/doc/ now!! 25
• 26

More Related Content

KEY
Coqによる証明駆動開発
Hiroki Mizuno
 
KEY
Coq to Rubyによる証明駆動開発@名古屋ruby会議02
Hiroki Mizuno
 
DOCX
Todos a aprender
iedoce
 
DOC
Proyecto sexualidad
iedoce
 
KEY
Parallel Computing in R
mickey24
 
PDF
Py "Baseball" Data入門〜サービス(と野球)を支えるデータ分析基盤 #monotarotech
Shinichi Nakagawa
 
PDF
katagaitai CTF workshop #10 AESに対する相関電力解析
trmr
 
PDF
次世代シーケンサのデータ解析 技術開発編
mickey24
 
Coqによる証明駆動開発
Hiroki Mizuno
 
Coq to Rubyによる証明駆動開発@名古屋ruby会議02
Hiroki Mizuno
 
Todos a aprender
iedoce
 
Proyecto sexualidad
iedoce
 
Parallel Computing in R
mickey24
 
Py "Baseball" Data入門〜サービス(と野球)を支えるデータ分析基盤 #monotarotech
Shinichi Nakagawa
 
katagaitai CTF workshop #10 AESに対する相関電力解析
trmr
 
次世代シーケンサのデータ解析 技術開発編
mickey24
 

Similar to Suffix Array@Solr勉強会 (20)

PDF
メドピアでの開発の裏側
Hiromichi Hirakawa
 
PDF
Monitoring MongoDB (MongoSV)
Boxed Ice
 
PDF
スマートフォン勉強会＠関東 #11 どう考えてもdisconなものをiPhoneに移植してみた
Taro Matsuzawa
 
PDF
20160708 データ処理のプラットフォームとしてのpython 札幌
Ryuji Tamagawa
 
KEY
『繋がり』を見る：　Cytoscapeと周辺ツールを使ったグラフデータ可視化入門
Keiichiro Ono
 
PDF
20161004 データ処理のプラットフォームとしてのpythonとpandas 東京
Ryuji Tamagawa
 
PDF
MongoDB - Monitoring and queueing
Boxed Ice
 
PDF
MongoDB - Monitoring & queueing
Boxed Ice
 
KEY
ひけらかし会（visualization)
moai kids
 
PDF
Cassandra 分散データベース
あしたのオープンソース研究所 　
 
PDF
三角関数の加法定理はなぜ難しいのか
鉄次 尾形
 
PDF
Monitoring MongoDB (MongoUK)
Boxed Ice
 
KEY
Programming Contest Hacks
Kosei Moriyama
 
PDF
Mpa Ji Study Presentaton
robcoulthard
 
PDF
[DL輪読会]Tracking Emerges by Colorizing Videos
Deep Learning JP
 
PDF
Google Polymer in Action
Jeongkyu Shin
 
PDF
DevLove k8s nobusue 20180711
Nobuhiro Sue
 
PDF
バ、バカな...！ハッカソンの中で成長しているだと...！？
Kenji Tanaka
 
PDF
SendaiRubyKaigi02 LT
Hiroshi SHIBATA
 
KEY
証明駆動開発のたのしみ@名古屋reject会議
Hiroki Mizuno
 
メドピアでの開発の裏側
Hiromichi Hirakawa
 
Monitoring MongoDB (MongoSV)
Boxed Ice
 
スマートフォン勉強会＠関東 #11 どう考えてもdisconなものをiPhoneに移植してみた
Taro Matsuzawa
 
20160708 データ処理のプラットフォームとしてのpython 札幌
Ryuji Tamagawa
 
『繋がり』を見る：　Cytoscapeと周辺ツールを使ったグラフデータ可視化入門
Keiichiro Ono
 
20161004 データ処理のプラットフォームとしてのpythonとpandas 東京
Ryuji Tamagawa
 
MongoDB - Monitoring and queueing
Boxed Ice
 
MongoDB - Monitoring & queueing
Boxed Ice
 
ひけらかし会（visualization)
moai kids
 
Cassandra 分散データベース
あしたのオープンソース研究所 　
 
三角関数の加法定理はなぜ難しいのか
鉄次 尾形
 
Monitoring MongoDB (MongoUK)
Boxed Ice
 
Programming Contest Hacks
Kosei Moriyama
 
Mpa Ji Study Presentaton
robcoulthard
 
[DL輪読会]Tracking Emerges by Colorizing Videos
Deep Learning JP
 
Google Polymer in Action
Jeongkyu Shin
 
DevLove k8s nobusue 20180711
Nobuhiro Sue
 
バ、バカな...！ハッカソンの中で成長しているだと...！？
Kenji Tanaka
 
SendaiRubyKaigi02 LT
Hiroshi SHIBATA
 
証明駆動開発のたのしみ@名古屋reject会議
Hiroki Mizuno
 
Ad

More from nobu_k (8)

PDF
Elasticsearchと機械学習を実際に連携させる
nobu_k
 
PDF
機械学習を利用したちょっとリッチな検索
nobu_k
 
PDF
4th PFI System reading
nobu_k
 
PDF
Goraft and InfluxDB
nobu_k
 
PPT
Transactional Information Systems入門
nobu_k
 
PDF
Riak Source Code Reading #2: Erlang Client
nobu_k
 
PPTX
Paxos
nobu_k
 
KEY
第一回MongoDBソースコードリーディング
nobu_k
 
Elasticsearchと機械学習を実際に連携させる
nobu_k
 
機械学習を利用したちょっとリッチな検索
nobu_k
 
4th PFI System reading
nobu_k
 
Goraft and InfluxDB
nobu_k
 
Transactional Information Systems入門
nobu_k
 
Riak Source Code Reading #2: Erlang Client
nobu_k
 
Paxos
nobu_k
 
第一回MongoDBソースコードリーディング
nobu_k
 
Ad

Suffix Array@Solr勉強会

  • 1. Suffix Array Solr 2011/12/19 1
  • 2. • (@nobu_k) • Preferred Infrastructure (PFI FI) • • • Sedue(2 ) 2
  • 3. Suffix Array • Suffix Array(SA): • ( ) 1 • Sedue • SA • • +Sedue • ” - ” 3
  • 4. • • • ( ) • n-gram(q-gram) • 4
  • 6. Suffix( ) 0: mississippi 1: ississippi 2: ssissippi 3: sissippi mississippi 4: issippi 5: ssippi 6: sippi 7: ippi 8: ppi 9: pi 10: i 6
  • 7. Suffix Array 0: mississippi 10: i 1: ississippi 7: ippi 2: ssissippi 4: issippi 3: sissippi 1: ississippi 4: issippi 0: mississippi 5: ssippi 9: pi 6: sippi 8: ppi 7: ippi 6: sippi 8: ppi 3: sissippi 9: pi 5: ssippi 10: i 2: ssissippi 7
  • 8. 10: i 7: ippi • mississippi ’si’ 4: issippi 1: ississippi • ’si’ 0: mississippi 9: pi • 8: ppi 6: 3: sippi sissippi • 5: ssippi 2: ssissippi • 3 6 8
  • 9. 10: i SA[i]: 7: ippi 4: issippi 10 7 4 1 0 9 8 6 3 5 2 1: ississippi T[i]: 0: mississippi 9: pi m i s s i s s i p p i 8: ppi 6: sippi 3: sissippi 6 5: ssippi T[SA[6]] 2: ssissippi → T[8] → “ppi” 9
  • 10. (1/3) T[i]: 1 2 3 ... n SA SA[i] 10
  • 11. (2/3) RedBull !! 1. RedBull *2 RedBull SA[i] 2. RedBull 1 2 3 ... n 11
  • 12. (3/3) 3. RedBull 1 2 3 ... n 4. ( 1, 3), ( 2, 4), ( 3, 2),...,( n, 2) 12
  • 13. • SA • + • /n-gram • SA • 13
  • 14. SA • (n-gram ) • • n-gram • • • “THIS IS IT” • proximity 14
  • 15. SA • • • • • HDD • ( ) • • 15
  • 16. • • ( ) • SAIS • • HDD • (dc3, dc7) • Sedue Haskell C++ • @tanakh++ 16
  • 17. • ( ) • • • 1 100GB/day • Sedue • SA n-gram • n-gram • SA n-gram • 17
  • 18. HDD • HDD • OK • • • SSD • SSD • Sedue 20 (80MB) • SA[i] 18
  • 19. VS 1. SA • 2. • SSD+ 500 3. • O(N) CPU 4. • • malloc 19
  • 20. • Sedue 1 56 • : 40 • : 16 (UTF-16) • 2 3 • • = • • SSD • 20
  • 21. SA • • 4(+1) • 2-gram • • % OK • • ” ” • 21
  • 22. • • • • • 22
  • 23. : groonga • Sedue groonga • • • Sedue groonga!! 23
  • 24. : • • • (http://jubat.us/) • http://github.com/jubatus • @JubatusOfficial • with NTT PF 24
  • 25. : Fluentd • Ruby • Treasure Data, Inc. • @frsyuki, @kzk_mover • Solr • gem install fluentd • Visit http://fluentd.org/doc/ now!! 25
  • 26. • 26

Editor's Notes