Abstract image of a woman sitting on books and studying on a laptop

эффективный поиск ближайших соседей в Postgre sql. о.бартунов. зал 2

R
rit2011

The document discusses techniques for efficient k-nearest neighbor searches in PostgreSQL. It begins by describing the kNN search problem and existing solutions. It then proposes modifying the generalized search tree (GiST) index in PostgreSQL to use a best-first search strategy rather than depth-first search, to avoid a full scan of the index and read tuples in the optimal order. The techniques aim to support different data types, tuple visibility, and efficient performance for large datasets. Examples demonstrating the approach on synthetic point data are also provided.

Technology
1 of 56
Downloaded 11 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
Эффективный поиск ближайших соседей в PostgreSQL Oleg Bartunov, Teodor Sigaev Sternberg Astronomical Institute, Moscow University
Knn-search: The problem ● What are interesting points near Royal Oak pub in Ottawa ? ● What are the closest events to the May 20, 2009 in Ottawa ? ● Similar images – feature extraction, Hamming distance ● Classification problem (major voting) ● ............ ● GIS, Science (high-dimensional data) ● Data merging (de-duplication) 24.04.11
Knn-search: Existing solutions knn=# select id, date, event from events order by date <-> '1957-10-04'::date asc limit 10; id | date | event --------+------------+----------------------------------------------------------------- 58137 | 1957-10-04 | U.S.S.R. launches Sputnik I, 1st artificial Earth satellite 58136 | 1957-10-04 | "Leave It to Beaver," debuts on CBS 117062 | 1957-10-04 | Gregory T Linteris, Demarest, New Jersey, astronaut, sk: STS 83 117061 | 1957-10-04 | Christina Smith, born in Miami, Florida, playmate, Mar, 1978 102670 | 1957-10-05 | Larry Saumell, jockey 31456 | 1957-10-03 | Willy Brandt elected mayor of West Berlin 58291 | 1957-10-05 | 12th Ryder Cup: Britain-Ireland, 7 -4 at Lindrick GC, England 58290 | 1957-10-05 | 11th NHL All-Star Game: All-Stars beat Montreal 5-3 at Montreal 58292 | 1957-10-05 | Yugoslav dissident Milovan Djilos sentenced to 7 years 102669 | 1957-10-05 | Jeanne Evert, tennis player, Chris' sister (10 rows) Time: 115.548 ms ● Very inefficient: ● Full table scan, btree index on date won't help. ● Sort full table 24.04.11
Knn-search: Existing solutions ● Find 10 closest points in table T MS SQL the best solution: Number is a table with integers http://blogs.msdn.com/b/isaac/archive/2008/10/23/nearest-neighbors.aspx DECLARE @start FLOAT = 1000; WITH NearestPoints AS ( SELECT TOP(1) WITH TIES *, T.g.STDistance(@x) AS dist FROM Numbers JOIN T WITH(INDEX(spatial_index)) ON T.g.STDistance(@x) < @start*POWER(2,Numbers.n) ORDER BY n ) SELECT TOP(10) * FROM NearestPoints ORDER BY n, dist Denali supports knn, but still lose the above trick :) http://www.sqlskills.com/BLOGS/BOBB/post/The-nearest-neighbor-optimization-in-SQL-Server- Denali.aspx 24.04.11
Knn-search: Existing solutions ● Oracle Spatial has SDO_NN function (only 2 dimensions !) http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14255/s do_operat.htm#i78067 SELECT /*+ INDEX(c cola_spatial_idx) */ c.mkt_id, c.name FROM cola_markets c WHERE SDO_NN(c.shape, sdo_geometry(2001, NULL, sdo_point_type(10,7,NULL), NULL, NULL), 'sdo_num_res=2') = 'TRUE'; 24.04.11
Knn-search: Existing solutions ● Traditional way to speedup query ● Use indexes - very inefficient (no search query !) – Scan full index – Full table scan, but in random order ! – Sort full table – Better not to use index at all ! ● Constrain data space (range search) – Incremental search → too many queries – Need to know in advance size of neighbourhood, how ? 1Km is ok for Paris, but too small for Siberia – Maintain 'density map' ? 24.04.11
What's a neighbourhood ? 24.04.11
Zoom 24.04.11
Knn-search: What do we want ! ● We want to avoid full table scan – read only <right> tuples ● So, we need index ● We want to avoid sorting – read <right> tuples in <right> order ● So, we need special strategy to traverse index ● We want to support tuples visibility ● So, we should be able to resume index traverse 24.04.11
R-tree index Root page Root page R1 I1 E R1 R2 E E E R2 I2 E Inner page 11 Inner page Inner page 22 Inner page I1 I3 I2 I4 I3 E E E E E E Leaf page 11 Leaf page Leaf page 33 Leaf page Leaf page 22 Leaf page I4 E E E E E E E E E E E E E 24.04.11
● Visualization of R-tree index using Gevel. ● Greece (data from rtreeportal.org) 24.04.11
R-tree index R1 I1 E SELECT E * E FROM E events R2 I2 E WHERE events.coord <@ 'QUERY'; I3 E E E E ● Root page: R1, R2 keys ● Inner pages: I3, I2 keys E ● Leaf pages: 4 points E Query I4 E E ● Very efficient for Search ! E E E 24.04.11
Knn-search: Index traverse ● Depth First Search (stack, LIFO) R-tree search ● Breadth First Search (queue, FIFO) 24.04.11 ● Both strategies are not good for us – full index scan
Knn-search: Index traverse ● Best First Search (PQ, priority queue). Maintain order of items in PQ according their distance from given point ● Distance to MBR (rectangle for Rtree) for internal pages – minimum distance of all items in that MBR ● Distance = 0 for MBR with given point ● Distance to point for leaf pages ● Each time we extract point from PQ we output it – it is next closest point ! If we extract rectangle, we expand it by pushing their children (rectangles and points) into the queue. ● We traverse index by visiting only interesting nodes ! 24.04.11
Knn-search: Index traverse ● Simple example – non-overlapped partitioning 24.04.11
Knn-search: Index traverse ● Simple example – non-overlapped partitioning ● Priority Queue ● 1: {1,2,3,4,5,6,7,8,9} ● 2: {2,5,6,7,9}, {1,3,4,8} ● 3: {5,6,7,9}, {1,3,4,8}, {2} ● 4: {5,9}, {1,3,4,8}, {2}, {6,7} ● 5: {1,3,4,8}, 5, {2}, {6,7}, 9 ● 6: {1,3,4}, {8}, 5, {2}, {6,7}, 9 ● 7: 4, {8}, 5, {2}, {6,7}, 3, 1, 9 we can output 4 without visit other rectangles ! ● 8: 5, {2}, {6,7}, 3, 8, 1, 9 ● 9: {6,7}, 3, 2, 8, 1, 9 ● 10: 3, 2, 8, 1, 9, 6, 7 24.04.11
Knn-search: Index traverse ● Simple example – non-overlapped partitioning ● Priority Queue ● 1: {1,2,3,4,5,6,7,8,9} ● 2: {2,5,6,7,9}, {1,3,4,8} ● 3: {5,6,7,9}, {1,3,4,8}, {2} ● 4: {5,9}, {1,3,4,8}, {2}, {6,7} ● 5: {1,3,4,8}, 5, {2}, {6,7}, 9 ● 6: {1,3,4}, {8}, 5, {2}, {6,7}, 9 ● 7: 4, {8}, 5, {2}, {6,7}, 3, 1, 9 ● 8: 5, {2}, {6,7}, 3, 8, 1, 9 24.04.11
Knn-search: Performance ● SEQ (no index) – base performance ● Sequentually read full table + Sort full table (can be very bad, sort_mem !) ● DFS – very bad ! ● Full index scan + Random read full table + Sort full table ● BFS – the best for small k ! ● Partial index scan + Random read k-records – T(index scan) ~ Height of Search tree ~ log(n) ● Performance win BFS/SEQ ~ Nrelpages/k, for small k. The more rows, the more benefit ! ● Can still win even for k=n (for large tables) - no sort ! 24.04.11
Knn-search: What do we want ! ● + We want to avoid full table scan – read only <right> tuples ● So, we need index ● + We want to avoid sorting – read <right> tuples in <right> order ● So, we need special strategy to traverse index ● + We want to support tuples visibility ● So, we should be able to resume index traverse ● We want to support many data types ● So, we need to modify GiST 24.04.11
Knn-search: modify GiST ● GiST – Generalized Search Tree, provides ● API to build custom disk-based search trees (any tree, where key of internal page is a Union of keys on children pages) ● Recovery and Concurrency ● Data type and query extendability ● GiST is widely used in GIS (PostGIS), text search,... ● Current strategy of search tree traverse is DFS ● Not good for knn-search ● We need to add Best First Search strategy for knn-search ● Retain API compatibility 24.04.11
Knn-search: syntax ● Knn-query uses ORDER BY clause SELECT … FROM … WHERE … ORDER BY p < - > '(0.,0.)'::point LIMIT k; < - > - distance operator, should be provided for data type 24.04.11
GiST interface ● compress/decompress ● same ● union ● penalty ● picksplit ● consistent ● distance - is needed if the operator class wishes to support ordered scans (nearest-neighbor searches) 24.04.11
Depth First Search Push BlkNo of root Go away into stack Y N Is stack empty? Push all matched N Pop BlkNo BlkNos into stack and read page Is a leaf? Return all matched entries Y 24.04.11
KNN-search: Priority Queue Go away Push (BlkNo of root, distance = 0) Y N Is queue empty? Read index page, push matched pairs Pop (pointer, distance) (pointer, distance) N Distance is a result of consistent methods Is a pointer to heap? Return pointer Y 24.04.11
GiST: Technical details ● Priority queue implemented as a RB-tree (Red-Black tree) ● Each node of RB-tree contains a list of pointers - pointers to internal pages follow pointers to heap. 10.3 Heap (10,12) HEAD Heap (34,1) 8.1 11.9 Heap (9,30) TAIL of Heap (98,11) heap's 9.0 pointers Index (p 34) Index (p 102) 24.04.11
GiST: Technical details Depth First Search Best First Search push Stack, Root; push PQ, Root; While Stack { While PQ { If p is heap { If p is heap { output p; output p; else { else { children = get_children(p); Children = get_children(p); push Stack, children; push PQ, children; } } } } ● For non-knn search all distances are zero, so PQ => Stack and BFS => DFS ● We can use only one strategy for both – normal search and knn- search ! 24.04.11
Knn-search: What do we want ! ● + We want to avoid full table scan – read only <right> tuples ● So, we need index ● + We want to avoid sorting – read <right> tuples in <right> order ● So, we need special strategy to traverse index ● + We want to support tuples visibility ● So, we should be able to resume index traverse ● + We want to support many data types ● So, we need to modify GiST 24.04.11
Knn-search: Examples ● Synthetic data – randomly distributed points create table qq ( id serial, p point, s int4); insert into qq (p,s) select point( p.lat, p.long), (random()*1000)::int from ( select (0.5-random())*180 as lat, random()*360 as long from ( select generate_series(1,1000000) ) as t ) as p; create index qq_p_s_idx on qq using gist(p); analyze qq; ● Query – find k-closest points to (0,0) set enable_indexscan=on|off; explain (analyze on, buffers on) select * from qq order by (p <-> '(0,0)') asc limit 10; 24.04.11
Knn-search: Examples ● postgresql.conf: shared_buffers = 512MB #32MB work_mem = 32MB #1MB maintenance_work_mem = 256MB #16MB checkpoint_segments = 16 effective_cache_size = 1GB #128MB ● Index statistics (n=1000,000) Number of levels: 3 Number of pages: 8787 Number of leaf pages: 8704 Number of tuples: 1008786 Number of invalid tuples: 0 Number of leaf tuples: 1000000 Total size of tuples: 44492028 bytes Total size of leaf tuples: 44104448 bytes Total size of index: 71983104 bytes 24.04.11
Knn-search: Examples k=1, n=1,000,000 Limit (cost=24853.00..24853.00 rows=1 width=24) (actual time=469.129..469.130 rows=1 loops=1) Buffers: shared hit=7353 -> Sort (cost=24853.00..27353.00 rows=1000000 width=24) (actual time=469.128..469.128 rows=1 loops=1) Sort Key: ((p <-> '(0,0)'::point)) Sort Method: top-N heapsort Memory: 25kB Buffers: shared hit=7353 -> Seq Scan on qq (cost=0.00..19853.00 rows=1000000 width=24) (actual time=0.007..241.539 rows=1000000 loops=1) Buffers: shared hit=7353 Total runtime: 469.150 ms Limit (cost=0.00..0.08 rows=1 width=24) (actual time=0.104..0.104 rows=1 loops=1) Buffers: shared hit=4 -> Index Scan using qq_p_idx on qq (cost=0.00..82060.60 rows=1000000 width=24) (actual time=0.104..0.104 rows=1 loops=1) Sort Cond: (p <-> '(0,0)'::point) Buffers: shared hit=4 Total runtime: 0.117 ms 4000 times faster ! -------------------------- 24.04.11
Knn-search: Examples n=1000,000 k :hit :knn : seq :sortmem --------------------------------------------------- 1 :4 :0.117 :469.150 : 25 10 :17 :0.289 :471.735 : 25 100 :118 :0.872 :468.244 : 32 1000 :1099 :7.107 :473.840 : 127 10000 :10234 :31.629 :525.557 : 1550 100000 :101159:321.182:994.925: 13957 24.04.11
Knn-search: Examples n=10,000 K :hit :knn : seq ---------------------------------- 1 :3 :0.117 :6.072 10 :13 :0.247 :5.014 100 :103 :0.295 :6.381 1000 :996 :1.605 :8.670 10000 :9916 :16.487 :14.706 -> knn lose if k=n, n is small 24.04.11
Knn-search: Examples ● Real data 2 mln points US, geonames 24.04.11
Knn-search: Examples ● Query: find 10 closest points in US with 'mars' in names to the point (5,5) - create composite index: create index pt_fts_idx on geo using gist(point, to_tsvector('english',asciiname)); =# explain (analyze on, buffers on) select asciiname,point, (point <-> '5.0,5.0'::point) as dist from geo where to_tsvector('english', asciiname) @@ to_tsquery('english','mars') order by dist asc limit 10; QUERY PLAN --------------------------------------------------------------------------------------- Limit (cost=0.00..33.55 rows=10 width=35) (actual time=0.452..0.597 rows=10 loops=1) Buffers: shared hit=56 -> Index Scan using pt_fts_idx on geo (cost=0.00..34313.91 rows=10227 width=35) (actual time=0.452..0.592 rows=10 loops=1) Index Cond: (to_tsvector('english'::regconfig, (asciiname)::text) @@ '''mar'''::tsquery) Sort Cond: (point <-> '(5,5)'::point) Buffers: shared hit=56 Total runtime: 0.629 ms (7 rows) 24.04.11
Knn-search: Existing solutions knn=# select id, date, event from events order by date <-> '1957-10-04'::date asc limit 10; id | date | event --------+------------+----------------------------------------------------------------- 58137 | 1957-10-04 | U.S.S.R. launches Sputnik I, 1st artificial Earth satellite 58136 | 1957-10-04 | "Leave It to Beaver," debuts on CBS 117062 | 1957-10-04 | Gregory T Linteris, Demarest, New Jersey, astronaut, sk: STS 83 117061 | 1957-10-04 | Christina Smith, born in Miami, Florida, playmate, Mar, 1978 102670 | 1957-10-05 | Larry Saumell, jockey 31456 | 1957-10-03 | Willy Brandt elected mayor of West Berlin 58291 | 1957-10-05 | 12th Ryder Cup: Britain-Ireland, 7 -4 at Lindrick GC, England 58290 | 1957-10-05 | 11th NHL All-Star Game: All-Stars beat Montreal 5-3 at Montreal 58292 | 1957-10-05 | Yugoslav dissident Milovan Djilos sentenced to 7 years 102669 | 1957-10-05 | Jeanne Evert, tennis player, Chris' sister (10 rows) Time: 115.548 ms ● Very inefficient: ● Full table scan, btree index on date won't help. ● Sort full table 24.04.11
Knn-search: Existing solutions contrib/btree_gist knn=# select id, date, event from events order by date <-> '1957-10-04'::date asc limit 10; id | date | event --------+------------+----------------------------------------------------------------- 58137 | 1957-10-04 | U.S.S.R. launches Sputnik I, 1st artificial Earth satellite 58136 | 1957-10-04 | "Leave It to Beaver," debuts on CBS 117062 | 1957-10-04 | Gregory T Linteris, Demarest, New Jersey, astronaut, sk: STS 83 117061 | 1957-10-04 | Christina Smith, born in Miami, Florida, playmate, Mar, 1978 102670 | 1957-10-05 | Larry Saumell, jockey 31456 | 1957-10-03 | Willy Brandt elected mayor of West Berlin 58291 | 1957-10-05 | 12th Ryder Cup: Britain-Ireland, 7 -4 at Lindrick GC, England 58290 | 1957-10-05 | 11th NHL All-Star Game: All-Stars beat Montreal 5-3 at Montreal 58292 | 1957-10-05 | Yugoslav dissident Milovan Djilos sentenced to 7 years 102669 | 1957-10-05 | Jeanne Evert, tennis player, Chris' sister (10 rows) Time: 0.590 ms ● Very inefficient: ● 8 index pages read + 10 tuples read ● NO sorting ● About 200 times faster ! 24.04.11
Knn-search: Examples ● pg_trgm support – distance = 1 – Similarity knn=# select date, event, ('jeorge ewashington' <-> event ) as dist from events order by dist asc limit 10; date | event | dist ------------+---------------------------------------------------+---------- 1732-02-11 | George Washington | 0.458333 1792-12-05 | George Washington re-elected U.S. pres | 0.674419 1811-02-23 | George Washington Hewitt, composer | 0.675 1753-08-04 | George Washington becomes a master mason | 0.697674 1941-07-19 | Jennifer Dunn, Rep-R-Washington | 0.710526 1945-05-12 | Jayotis Washington, rocker | 0.714286 1817-05-05 | George Washington Julian, MC, Union, died in 1899 | 0.72549 1789-08-25 | Mary Ball Washington, mother of George, dies | 0.729167 1844-01-12 | George Washington Cable, American Novelist | 0.729167 1925-01-31 | George Washington Cable, American Novelist | 0.729167 (10 rows) Time: 187.604 ms 24.04.11
Knn-search: Examples ● Corner case for knn-search - all data are on the same distance from point Q ! Q 24.04.11
Knn-search: Examples ● Corner case for Best First Strategy - all data are on the same distance from point Q ! create table circle (id serial, p point, s int4); insert into circle (p,s) select point( p.x, p.y), (random()*1000)::int from ( select t.x, sqrt(1- t.x*t.x) as y from ( select random() as x, generate_series(1,1000000) ) as t ) as p; create index circle_p_idx on circle using gist(p); analyze circle; Number of levels: 3 Number of pages: 8266 Number of leaf pages: 8201 24.04.11
Knn-search: Examples ● Corner case for knn-search - all data are on the same distance from point Q ! =# explain (analyze on, buffers on) select * from circle order by (p <-> '(0,0)') asc limit 10; Limit (cost=0.00..0.80 rows=10 width=24) (actual time=226.907..226.924 rows=10 loops=1) Buffers: shared hit=8276 -> Index Scan using circle_p_idx on circle (cost=0.00..79976.58 rows=1000000 width=24) (actual time=226.905..226.921 rows=10 loops=1) Sort Cond: (p <-> '(0,0)'::point) Buffers: shared hit=8276 – read all index Total runtime: 230.885 ms ● Still 2 times faster than SEQ (454.331 ms) because of sorting 24.04.11
Knn-search: Status Committed to 9.1 24.04.11
Knn: History of development ● Начало проекта - Sep 8, 2007 at 7:54 PM date Sat, Sep 8, 2007 at 7:54 PM subject Chat with Sergey V. Karpov 7:36 PM me: я тут knn-search занимаюсь, масса интересного. Все думаю, как в постгресе это поиметь Sergey: а что это такое? 7:37 PM me: k-nearest соседей - супер важная задача найти 5 ближайших точек 7:38 PM Sergey: ближайших к чему? me: к заданной точке 7:39 PM Sergey: в какой системе координат? me: в любой, в n-мерном пространстве. В простом варианте - хотя бы на земле/небе 7:40 PM это нужно для поиска похожих картинок, например. навиный вариант повторять запросы - не катит 24.04.11
Knn: History of development ● TODO (http://www.sai.msu.su/~megera/wiki/TODO) начало 2008 года, уже есть понимание что делать ● 25 июля 2009 года – письмо Paul Ramsey (POSTGIS) ● 10 июля 2009 года – контракт Open Planning Project Inc. ● 20 ноября 2009 года – патч KNNGiST v.0.1 (модуль расш) ● Commitfest nightmare ● 22 июля 2010 – KNNGiST (v.0.8), commitfest ● 13 сентября 2010 – KNNGiST (v.0.9) ● 03 декабря 2010 – Tom Lane committed for 9.1 ! ● 21 января 2011 – contrib/btree_gist committed ! 24.04.11
Knn: History of development ● Итого: На проект ушло больше 3 лет ! ● Реальное программирование заняло несколько месяцев ● Основные причины: ● Отсутствие поддержки – Слабая информационная связь разработчиков и заказчиков ● Занятость разработчиков ● Усложнение процедуры рассмотрения проектов в сообществе 24.04.11
PostgreSQL 9.1 Что планировалось ? 24.04.11
Major features PostgreSQL 9.1 Совещание разработчиков в Оттаве, май 2010 24.04.11
Priorities for 9.1 http://wiki.postgresql.org/wiki/PgCon_2010_Developer_Meeting 24.04.11
PostgreSQL 9.1 Что удалось реализовать (основные новые возможности) 24.04.11
PostgreSQL 9.1 ● Функциональность: ● Synchronous Replication ● Per-column collation ● Unlogged tables ● Новинки ● SSI (Serializable Snapshot Isolation), статья Michael J. Cahill, Uwe Röhm, Alan D. Fekete. Serializable Isolation for Snapshot Databases. In the Proceedings of the 2008 ACM SIGMOD international conference on management of data, pages 729-738 24.04.11
PostgreSQL 9.1: SSI ● До 9.1 ● READ UNCOMMITTED=READ COMMITTED ● REPEATABLE READ=SERIALIZABLE, в текущей транзакции видны все записи зафиксированные до первого запроса или до выполнения команды, изменяющей данные, в рамках этой транзакции. ● 9.1 ● REPEATABLE READ ● SERIALIZABLE, гарантирует последовательное выполнение транзакций - как будто транзакции выполняются последовательно друг за другом. 24.04.11
PostgreSQL 9.1 ● Новинки: ● KNNGIST – поиск ближайших соседей, первая реализация для продакшн системы. ● SE/PostgreS - поддержка мандатной политики доступа, совместимой с SE Linux,в виде расширения, который вводит дополнительный слой проверки безопасности на основе использования меток безопасности (security labels). ● wCTE (writable Common Table Expressions) - использование команд, которые изменяют данные (INSERT,UPDATE,DELETE), с оператором WITH. Это позволяет исполнять несколько разных операций в одном запросе. 24.04.11
PostgreSQL 9.1: wCTE Получить сводку удаленных постов по пользователю и обновить статистику пользователей. WITH deleted_posts AS ( DELETE FROM posts WHERE created < now() - '6 months'::INTERVAL RETURNING * ), deleted_per_user as ( SELECT user_id, count(*) FROM deleted_posts GROUP BY 1 ) UPDATE counts SET posts_count = posts_count - d.count FROM deleted_per_user d WHERE d.user_id = counts.user_id; 24.04.11
PostgreSQL 9.1 ● Расширяемость: ● Extensions - инфраструктура управления расширениями. Команды CREATE/DROP/ALTER EXTENSION позволяют манипулировать расширениями (устанавливать, обновлять, изменять ) как единым целым. ● SQL/MED - поддержка SQL стандарта в части управления внешними (вне базы данных) данными (Management of External Data) для обеспечения прозрачной интеграции со внешними данными. 24.04.11
PostgreSQL 9.1: Extensions postgres=# CREATE EXTENSION file_fdw; CREATE EXTENSION postgres=# dx List of installed extensions Name | Version | Schema | Description ----------+---------+------------+------------------------------------------- file_fdw | 1.0 | public | foreign-data wrapper for flat file access plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language (2 rows) postgres=# dx+ file_fdw Objects in extension "file_fdw" Object Description ----------------------------------------- foreign-data wrapper file_fdw function file_fdw_handler() function file_fdw_validator(text[],oid) (3 rows) 24.04.11
PostgreSQL 9.1: SQL/MED postgres=# CREATE SERVER file_server FOREIGN DATA WRAPPER file_fdw; postgres=# CREATE FOREIGN TABLE passwd ( username text, pass text, uid int4, gid int4, gecos text, home text, shell text ) SERVER file_server OPTIONS (format 'text', filename '/etc/passwd', delimiter ':', null ''); postgres=# select * from passwd order by uid asc limit 3; username | pass | uid | gid | gecos | home | shell ----------+------+-----+-----+--------+-----------+----------- root | x | 0 | 0 | root | /root | /bin/bash daemon | x | 1 | 1 | daemon | /usr/sbin | /bin/sh bin | x | 2 | 2 | bin | /bin | /bin/sh (3 rows) postgres=# d List of relations Schema | Name | Type | Owner --------+--------+---------------+---------- public | passwd | foreign table | postgres public | test | table | postgres (2 rows) 24.04.11
PostgreSQL 9.1 Полный список изменений http://developer.postgresql.org/pgdocs/postgres/release-9-1.html 24.04.11

More Related Content

PPT
как объяснить заказчику, что он не прав. денис тучин. зал 3
rit2011
 
PDF
полмиллиона юзеров в онлайне без падений оптимизация высоконагруженной Server...
rit2011
 
PPTX
Solaris 11 новые возможности. а. павленко. зал 2
rit2011
 
ODP
распределенное файловое хранилище (Nginx, zfs, perl). перепелица мамонтов. зал 2
rit2011
 
PDF
научные вызовы к субд и система Sci db. п. велихов. зал 2
rit2011
 
PDF
Chef. кто на кухне хозяин. концепция devops. а,титов. зал 2
rit2011
 
PPTX
типология личности и прогноз отношений по а. афанасьеву. сергей котырев. зал 2
rit2011
 
PPT
как объяснить заказчику, что он не прав. денис тучин. зал 3
rit2011
 
как объяснить заказчику, что он не прав. денис тучин. зал 3
rit2011
 
полмиллиона юзеров в онлайне без падений оптимизация высоконагруженной Server...
rit2011
 
Solaris 11 новые возможности. а. павленко. зал 2
rit2011
 
распределенное файловое хранилище (Nginx, zfs, perl). перепелица мамонтов. зал 2
rit2011
 
научные вызовы к субд и система Sci db. п. велихов. зал 2
rit2011
 
Chef. кто на кухне хозяин. концепция devops. а,титов. зал 2
rit2011
 
типология личности и прогноз отношений по а. афанасьеву. сергей котырев. зал 2
rit2011
 
как объяснить заказчику, что он не прав. денис тучин. зал 3
rit2011
 

Viewers also liked (10)

PDF
распределенное файловое хранилище (Nginx, zfs, perl). перепелица мамонтов. зал 2
rit2011
 
PPTX
использование Solaris zones для решения различных задач. а. павленко. зал 4
rit2011
 
PPT
поиск узких мест в производительности My sql ботанический определитель. г. ру...
rit2011
 
PPTX
менеджмент в стиле коучинг. с. панкратов. зал 2
rit2011
 
PDF
I pv6 малоизвестные подробности. андрей пантюхин. зал 2
rit2011
 
PDF
мои модули и патчи для Nginx. максим дунин. зал 1
rit2011
 
PDF
бэм! в.харисов, с. бережной. зал 2
rit2011
 
PDF
классификация Ddos. александр лямин, артем гавриченков. зал 2
rit2011
 
PDF
масштабируемый Sphinx кластер. вячеслав крюков. зал 1
rit2011
 
POTX
рисуем тз. эффективный способ коммуникации в веб проектах. артем вольфтруб. з...
rit2011
 
распределенное файловое хранилище (Nginx, zfs, perl). перепелица мамонтов. зал 2
rit2011
 
использование Solaris zones для решения различных задач. а. павленко. зал 4
rit2011
 
поиск узких мест в производительности My sql ботанический определитель. г. ру...
rit2011
 
менеджмент в стиле коучинг. с. панкратов. зал 2
rit2011
 
I pv6 малоизвестные подробности. андрей пантюхин. зал 2
rit2011
 
мои модули и патчи для Nginx. максим дунин. зал 1
rit2011
 
бэм! в.харисов, с. бережной. зал 2
rit2011
 
классификация Ddos. александр лямин, артем гавриченков. зал 2
rit2011
 
масштабируемый Sphinx кластер. вячеслав крюков. зал 1
rit2011
 
рисуем тз. эффективный способ коммуникации в веб проектах. артем вольфтруб. з...
rit2011
 
Ad

Similar to эффективный поиск ближайших соседей в Postgre sql. о.бартунов. зал 2 (20)

PPT
Seq db searching
Er Puspendra Tripathi
 
KEY
2011.06.20 stratified-btree
Acunu
 
PPT
Dic hash
ppreeta
 
PPTX
Minimum spanning Tree
Narendra Singh Patel
 
PPT
Multidimensional Indexing
Digvijay Singh
 
PDF
Lecture14 - Advanced topics in association rules
Albert Orriols-Puig
 
PPT
B-tree & R-tree
Shakil Ahmed
 
PPTX
Alignment of raw reads in Avadis NGS
Strand Life Sciences Pvt Ltd
 
KEY
Verification with LoLA: 7 Implementation
Universität Rostock
 
PDF
E1071
taikhoan262
 
PDF
Context-aware Mobile Recommendation Services for Conference Participants
Ralf Klamma
 
PDF
Dmss2011 public
Yasuo Tabei
 
PPT
Graph
Shakil Ahmed
 
PDF
EMF Compare 2.0: Scaling to Millions (updated)
mikaelbarbero
 
PDF
Cassandra on Castle
Acunu
 
PDF
Gwt sdm public
Yasuo Tabei
 
PDF
ISWC 2012 "Efficient execution of top-k SPARQL queries"
Sara Magliacane
 
PPTX
Bioinformatics t5-databasesearching v2014
Prof. Wim Van Criekinge
 
PDF
302 B+Tree Ind Hash
VJ Aiswaryadevi
 
PPTX
2015 bioinformatics database_searching_wimvancriekinge
Prof. Wim Van Criekinge
 
Seq db searching
Er Puspendra Tripathi
 
2011.06.20 stratified-btree
Acunu
 
Dic hash
ppreeta
 
Minimum spanning Tree
Narendra Singh Patel
 
Multidimensional Indexing
Digvijay Singh
 
Lecture14 - Advanced topics in association rules
Albert Orriols-Puig
 
B-tree & R-tree
Shakil Ahmed
 
Alignment of raw reads in Avadis NGS
Strand Life Sciences Pvt Ltd
 
Verification with LoLA: 7 Implementation
Universität Rostock
 
E1071
taikhoan262
 
Context-aware Mobile Recommendation Services for Conference Participants
Ralf Klamma
 
Dmss2011 public
Yasuo Tabei
 
Graph
Shakil Ahmed
 
EMF Compare 2.0: Scaling to Millions (updated)
mikaelbarbero
 
Cassandra on Castle
Acunu
 
Gwt sdm public
Yasuo Tabei
 
ISWC 2012 "Efficient execution of top-k SPARQL queries"
Sara Magliacane
 
Bioinformatics t5-databasesearching v2014
Prof. Wim Van Criekinge
 
302 B+Tree Ind Hash
VJ Aiswaryadevi
 
2015 bioinformatics database_searching_wimvancriekinge
Prof. Wim Van Criekinge
 
Ad

More from rit2011 (19)

PPT
классификация Ddos. александр лямин, артем гавриченков. зал 2
rit2011
 
PPTX
Kpi разработчика vs kpi разработки. евгения фирсова. зал 1
rit2011
 
PDF
ускорение Front end разработки с помощью haml, sass и compass. андрей ситник....
rit2011
 
PDF
ускорение Front end разработки с помощью haml, sass и compass. андрей ситник....
rit2011
 
PPT
что и почему вы должны программировать на Erlang.максим лапшин. зал 4
rit2011
 
PPT
безопасность веб приложений сегодня. дмитрий евтеев. зал 4
rit2011
 
PDF
как стать хорошим веб технологом. нарек мкртчян. зал 4
rit2011
 
PPTX
сотни серверов, десятки компонент. автоматизация раскладки и конфигурирования...
rit2011
 
PPT
выращиваем интерфейс своими руками. ольга павлова. зал 3
rit2011
 
PPT
от Flash к html5. александр бацуев. зал 4
rit2011
 
PPTX
Ie9 и ie10. алекс могилевский. зал 2
rit2011
 
PPTX
сотни серверов, десятки компонент. автоматизация раскладки и конфигурирования...
rit2011
 
PPTX
круглый стол по найму. александр зиза. зал 2
rit2011
 
PPTX
круглый стол по найму. александр зиза. зал 2
rit2011
 
PPT
как объяснить заказчику, что он не прав. денис тучин. зал 3
rit2011
 
PDF
как написать масштабируемую баннерокрутилку. денис бирюков, артем гавриченков...
rit2011
 
PDF
способы защиты медиа контента. возможен ли Drm в вебе. денис елданди, алексан...
rit2011
 
PDF
модульное тестирование для Perl. алексей шруб. зал 4
rit2011
 
PDF
автоматизированная сборка Flash приложений (as2, as3). андрей жданов. зал 4
rit2011
 
классификация Ddos. александр лямин, артем гавриченков. зал 2
rit2011
 
Kpi разработчика vs kpi разработки. евгения фирсова. зал 1
rit2011
 
ускорение Front end разработки с помощью haml, sass и compass. андрей ситник....
rit2011
 
ускорение Front end разработки с помощью haml, sass и compass. андрей ситник....
rit2011
 
что и почему вы должны программировать на Erlang.максим лапшин. зал 4
rit2011
 
безопасность веб приложений сегодня. дмитрий евтеев. зал 4
rit2011
 
как стать хорошим веб технологом. нарек мкртчян. зал 4
rit2011
 
сотни серверов, десятки компонент. автоматизация раскладки и конфигурирования...
rit2011
 
выращиваем интерфейс своими руками. ольга павлова. зал 3
rit2011
 
от Flash к html5. александр бацуев. зал 4
rit2011
 
Ie9 и ie10. алекс могилевский. зал 2
rit2011
 
сотни серверов, десятки компонент. автоматизация раскладки и конфигурирования...
rit2011
 
круглый стол по найму. александр зиза. зал 2
rit2011
 
круглый стол по найму. александр зиза. зал 2
rit2011
 
как объяснить заказчику, что он не прав. денис тучин. зал 3
rit2011
 
как написать масштабируемую баннерокрутилку. денис бирюков, артем гавриченков...
rit2011
 
способы защиты медиа контента. возможен ли Drm в вебе. денис елданди, алексан...
rit2011
 
модульное тестирование для Perl. алексей шруб. зал 4
rit2011
 
автоматизированная сборка Flash приложений (as2, as3). андрей жданов. зал 4
rit2011
 

Recently uploaded (20)

PDF
Developing a website for English-speaking practice to English as a foreign la...
IAESIJAI
 
PDF
Comparative analysis of machine learning models for fake news detection in so...
IAESIJAI
 
PDF
“A New Era of 3D Sensing: Transforming Industries and Creating Opportunities,...
Edge AI and Vision Alliance
 
PPTX
GROUP4NURSINGINFORMATICSREPORT-2 PRESENTATION
ClarisejaneSartillo1
 
PDF
Five Habits of High-Impact Board Members
OnBoard
 
PDF
A contest of sentiment analysis: k-nearest neighbor versus neural network
IAESIJAI
 
PPTX
Benefits of Physical activity for teenagers.pptx
Nokwanda Thabethe
 
PDF
A review of recent deep learning applications in wood surface defect identifi...
IAESIJAI
 
PDF
OpenACC and Open Hackathons Monthly Highlights July 2025
OpenACC
 
PDF
Enhancing plagiarism detection using data pre-processing and machine learning...
IAESIJAI
 
PPTX
TEXTILE technology diploma scope and career opportunities
kriti38
 
PPTX
AI IN MARKETING- PRESENTED BY ANWAR KABIR 1st June 2025.pptx
HiraJay1
 
PDF
Getting started with AI Agents and Multi-Agent Systems
Maxim Salnikov
 
PPTX
Custom Battery Pack Design Considerations for Performance and Safety
Epec Engineered Technologies
 
PDF
CloudStack 4.21: First Look Webinar slides
ShapeBlue
 
PPTX
Modernising the Digital Integration Hub
Daniel Toomey
 
PDF
Credit Without Borders: AI and Financial Inclusion in Bangladesh
Ehsan Tanim
 
PDF
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
PPTX
Microsoft Excel 365/2024 Beginner's training
frank yeboah
 
PPTX
Build Your First AI Agent with UiPath.pptx
suhanisingh58689
 
Developing a website for English-speaking practice to English as a foreign la...
IAESIJAI
 
Comparative analysis of machine learning models for fake news detection in so...
IAESIJAI
 
“A New Era of 3D Sensing: Transforming Industries and Creating Opportunities,...
Edge AI and Vision Alliance
 
GROUP4NURSINGINFORMATICSREPORT-2 PRESENTATION
ClarisejaneSartillo1
 
Five Habits of High-Impact Board Members
OnBoard
 
A contest of sentiment analysis: k-nearest neighbor versus neural network
IAESIJAI
 
Benefits of Physical activity for teenagers.pptx
Nokwanda Thabethe
 
A review of recent deep learning applications in wood surface defect identifi...
IAESIJAI
 
OpenACC and Open Hackathons Monthly Highlights July 2025
OpenACC
 
Enhancing plagiarism detection using data pre-processing and machine learning...
IAESIJAI
 
TEXTILE technology diploma scope and career opportunities
kriti38
 
AI IN MARKETING- PRESENTED BY ANWAR KABIR 1st June 2025.pptx
HiraJay1
 
Getting started with AI Agents and Multi-Agent Systems
Maxim Salnikov
 
Custom Battery Pack Design Considerations for Performance and Safety
Epec Engineered Technologies
 
CloudStack 4.21: First Look Webinar slides
ShapeBlue
 
Modernising the Digital Integration Hub
Daniel Toomey
 
Credit Without Borders: AI and Financial Inclusion in Bangladesh
Ehsan Tanim
 
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
Microsoft Excel 365/2024 Beginner's training
frank yeboah
 
Build Your First AI Agent with UiPath.pptx
suhanisingh58689
 

эффективный поиск ближайших соседей в Postgre sql. о.бартунов. зал 2

  • 1. Эффективный поиск ближайших соседей в PostgreSQL Oleg Bartunov, Teodor Sigaev Sternberg Astronomical Institute, Moscow University
  • 2. Knn-search: The problem ● What are interesting points near Royal Oak pub in Ottawa ? ● What are the closest events to the May 20, 2009 in Ottawa ? ● Similar images – feature extraction, Hamming distance ● Classification problem (major voting) ● ............ ● GIS, Science (high-dimensional data) ● Data merging (de-duplication) 24.04.11
  • 3. Knn-search: Existing solutions knn=# select id, date, event from events order by date <-> '1957-10-04'::date asc limit 10; id | date | event --------+------------+----------------------------------------------------------------- 58137 | 1957-10-04 | U.S.S.R. launches Sputnik I, 1st artificial Earth satellite 58136 | 1957-10-04 | "Leave It to Beaver," debuts on CBS 117062 | 1957-10-04 | Gregory T Linteris, Demarest, New Jersey, astronaut, sk: STS 83 117061 | 1957-10-04 | Christina Smith, born in Miami, Florida, playmate, Mar, 1978 102670 | 1957-10-05 | Larry Saumell, jockey 31456 | 1957-10-03 | Willy Brandt elected mayor of West Berlin 58291 | 1957-10-05 | 12th Ryder Cup: Britain-Ireland, 7 -4 at Lindrick GC, England 58290 | 1957-10-05 | 11th NHL All-Star Game: All-Stars beat Montreal 5-3 at Montreal 58292 | 1957-10-05 | Yugoslav dissident Milovan Djilos sentenced to 7 years 102669 | 1957-10-05 | Jeanne Evert, tennis player, Chris' sister (10 rows) Time: 115.548 ms ● Very inefficient: ● Full table scan, btree index on date won't help. ● Sort full table 24.04.11
  • 4. Knn-search: Existing solutions ● Find 10 closest points in table T MS SQL the best solution: Number is a table with integers http://blogs.msdn.com/b/isaac/archive/2008/10/23/nearest-neighbors.aspx DECLARE @start FLOAT = 1000; WITH NearestPoints AS ( SELECT TOP(1) WITH TIES *, T.g.STDistance(@x) AS dist FROM Numbers JOIN T WITH(INDEX(spatial_index)) ON T.g.STDistance(@x) < @start*POWER(2,Numbers.n) ORDER BY n ) SELECT TOP(10) * FROM NearestPoints ORDER BY n, dist Denali supports knn, but still lose the above trick :) http://www.sqlskills.com/BLOGS/BOBB/post/The-nearest-neighbor-optimization-in-SQL-Server- Denali.aspx 24.04.11
  • 5. Knn-search: Existing solutions ● Oracle Spatial has SDO_NN function (only 2 dimensions !) http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14255/s do_operat.htm#i78067 SELECT /*+ INDEX(c cola_spatial_idx) */ c.mkt_id, c.name FROM cola_markets c WHERE SDO_NN(c.shape, sdo_geometry(2001, NULL, sdo_point_type(10,7,NULL), NULL, NULL), 'sdo_num_res=2') = 'TRUE'; 24.04.11
  • 6. Knn-search: Existing solutions ● Traditional way to speedup query ● Use indexes - very inefficient (no search query !) – Scan full index – Full table scan, but in random order ! – Sort full table – Better not to use index at all ! ● Constrain data space (range search) – Incremental search → too many queries – Need to know in advance size of neighbourhood, how ? 1Km is ok for Paris, but too small for Siberia – Maintain 'density map' ? 24.04.11
  • 9. Knn-search: What do we want ! ● We want to avoid full table scan – read only <right> tuples ● So, we need index ● We want to avoid sorting – read <right> tuples in <right> order ● So, we need special strategy to traverse index ● We want to support tuples visibility ● So, we should be able to resume index traverse 24.04.11
  • 10. R-tree index Root page Root page R1 I1 E R1 R2 E E E R2 I2 E Inner page 11 Inner page Inner page 22 Inner page I1 I3 I2 I4 I3 E E E E E E Leaf page 11 Leaf page Leaf page 33 Leaf page Leaf page 22 Leaf page I4 E E E E E E E E E E E E E 24.04.11
  • 11. ● Visualization of R-tree index using Gevel. ● Greece (data from rtreeportal.org) 24.04.11
  • 12. R-tree index R1 I1 E SELECT E * E FROM E events R2 I2 E WHERE events.coord <@ 'QUERY'; I3 E E E E ● Root page: R1, R2 keys ● Inner pages: I3, I2 keys E ● Leaf pages: 4 points E Query I4 E E ● Very efficient for Search ! E E E 24.04.11
  • 13. Knn-search: Index traverse ● Depth First Search (stack, LIFO) R-tree search ● Breadth First Search (queue, FIFO) 24.04.11 ● Both strategies are not good for us – full index scan
  • 14. Knn-search: Index traverse ● Best First Search (PQ, priority queue). Maintain order of items in PQ according their distance from given point ● Distance to MBR (rectangle for Rtree) for internal pages – minimum distance of all items in that MBR ● Distance = 0 for MBR with given point ● Distance to point for leaf pages ● Each time we extract point from PQ we output it – it is next closest point ! If we extract rectangle, we expand it by pushing their children (rectangles and points) into the queue. ● We traverse index by visiting only interesting nodes ! 24.04.11
  • 15. Knn-search: Index traverse ● Simple example – non-overlapped partitioning 24.04.11
  • 16. Knn-search: Index traverse ● Simple example – non-overlapped partitioning ● Priority Queue ● 1: {1,2,3,4,5,6,7,8,9} ● 2: {2,5,6,7,9}, {1,3,4,8} ● 3: {5,6,7,9}, {1,3,4,8}, {2} ● 4: {5,9}, {1,3,4,8}, {2}, {6,7} ● 5: {1,3,4,8}, 5, {2}, {6,7}, 9 ● 6: {1,3,4}, {8}, 5, {2}, {6,7}, 9 ● 7: 4, {8}, 5, {2}, {6,7}, 3, 1, 9 we can output 4 without visit other rectangles ! ● 8: 5, {2}, {6,7}, 3, 8, 1, 9 ● 9: {6,7}, 3, 2, 8, 1, 9 ● 10: 3, 2, 8, 1, 9, 6, 7 24.04.11
  • 17. Knn-search: Index traverse ● Simple example – non-overlapped partitioning ● Priority Queue ● 1: {1,2,3,4,5,6,7,8,9} ● 2: {2,5,6,7,9}, {1,3,4,8} ● 3: {5,6,7,9}, {1,3,4,8}, {2} ● 4: {5,9}, {1,3,4,8}, {2}, {6,7} ● 5: {1,3,4,8}, 5, {2}, {6,7}, 9 ● 6: {1,3,4}, {8}, 5, {2}, {6,7}, 9 ● 7: 4, {8}, 5, {2}, {6,7}, 3, 1, 9 ● 8: 5, {2}, {6,7}, 3, 8, 1, 9 24.04.11
  • 18. Knn-search: Performance ● SEQ (no index) – base performance ● Sequentually read full table + Sort full table (can be very bad, sort_mem !) ● DFS – very bad ! ● Full index scan + Random read full table + Sort full table ● BFS – the best for small k ! ● Partial index scan + Random read k-records – T(index scan) ~ Height of Search tree ~ log(n) ● Performance win BFS/SEQ ~ Nrelpages/k, for small k. The more rows, the more benefit ! ● Can still win even for k=n (for large tables) - no sort ! 24.04.11
  • 19. Knn-search: What do we want ! ● + We want to avoid full table scan – read only <right> tuples ● So, we need index ● + We want to avoid sorting – read <right> tuples in <right> order ● So, we need special strategy to traverse index ● + We want to support tuples visibility ● So, we should be able to resume index traverse ● We want to support many data types ● So, we need to modify GiST 24.04.11
  • 20. Knn-search: modify GiST ● GiST – Generalized Search Tree, provides ● API to build custom disk-based search trees (any tree, where key of internal page is a Union of keys on children pages) ● Recovery and Concurrency ● Data type and query extendability ● GiST is widely used in GIS (PostGIS), text search,... ● Current strategy of search tree traverse is DFS ● Not good for knn-search ● We need to add Best First Search strategy for knn-search ● Retain API compatibility 24.04.11
  • 21. Knn-search: syntax ● Knn-query uses ORDER BY clause SELECT … FROM … WHERE … ORDER BY p < - > '(0.,0.)'::point LIMIT k; < - > - distance operator, should be provided for data type 24.04.11
  • 22. GiST interface ● compress/decompress ● same ● union ● penalty ● picksplit ● consistent ● distance - is needed if the operator class wishes to support ordered scans (nearest-neighbor searches) 24.04.11
  • 23. Depth First Search Push BlkNo of root Go away into stack Y N Is stack empty? Push all matched N Pop BlkNo BlkNos into stack and read page Is a leaf? Return all matched entries Y 24.04.11
  • 24. KNN-search: Priority Queue Go away Push (BlkNo of root, distance = 0) Y N Is queue empty? Read index page, push matched pairs Pop (pointer, distance) (pointer, distance) N Distance is a result of consistent methods Is a pointer to heap? Return pointer Y 24.04.11
  • 25. GiST: Technical details ● Priority queue implemented as a RB-tree (Red-Black tree) ● Each node of RB-tree contains a list of pointers - pointers to internal pages follow pointers to heap. 10.3 Heap (10,12) HEAD Heap (34,1) 8.1 11.9 Heap (9,30) TAIL of Heap (98,11) heap's 9.0 pointers Index (p 34) Index (p 102) 24.04.11
  • 26. GiST: Technical details Depth First Search Best First Search push Stack, Root; push PQ, Root; While Stack { While PQ { If p is heap { If p is heap { output p; output p; else { else { children = get_children(p); Children = get_children(p); push Stack, children; push PQ, children; } } } } ● For non-knn search all distances are zero, so PQ => Stack and BFS => DFS ● We can use only one strategy for both – normal search and knn- search ! 24.04.11
  • 27. Knn-search: What do we want ! ● + We want to avoid full table scan – read only <right> tuples ● So, we need index ● + We want to avoid sorting – read <right> tuples in <right> order ● So, we need special strategy to traverse index ● + We want to support tuples visibility ● So, we should be able to resume index traverse ● + We want to support many data types ● So, we need to modify GiST 24.04.11
  • 28. Knn-search: Examples ● Synthetic data – randomly distributed points create table qq ( id serial, p point, s int4); insert into qq (p,s) select point( p.lat, p.long), (random()*1000)::int from ( select (0.5-random())*180 as lat, random()*360 as long from ( select generate_series(1,1000000) ) as t ) as p; create index qq_p_s_idx on qq using gist(p); analyze qq; ● Query – find k-closest points to (0,0) set enable_indexscan=on|off; explain (analyze on, buffers on) select * from qq order by (p <-> '(0,0)') asc limit 10; 24.04.11
  • 29. Knn-search: Examples ● postgresql.conf: shared_buffers = 512MB #32MB work_mem = 32MB #1MB maintenance_work_mem = 256MB #16MB checkpoint_segments = 16 effective_cache_size = 1GB #128MB ● Index statistics (n=1000,000) Number of levels: 3 Number of pages: 8787 Number of leaf pages: 8704 Number of tuples: 1008786 Number of invalid tuples: 0 Number of leaf tuples: 1000000 Total size of tuples: 44492028 bytes Total size of leaf tuples: 44104448 bytes Total size of index: 71983104 bytes 24.04.11
  • 30. Knn-search: Examples k=1, n=1,000,000 Limit (cost=24853.00..24853.00 rows=1 width=24) (actual time=469.129..469.130 rows=1 loops=1) Buffers: shared hit=7353 -> Sort (cost=24853.00..27353.00 rows=1000000 width=24) (actual time=469.128..469.128 rows=1 loops=1) Sort Key: ((p <-> '(0,0)'::point)) Sort Method: top-N heapsort Memory: 25kB Buffers: shared hit=7353 -> Seq Scan on qq (cost=0.00..19853.00 rows=1000000 width=24) (actual time=0.007..241.539 rows=1000000 loops=1) Buffers: shared hit=7353 Total runtime: 469.150 ms Limit (cost=0.00..0.08 rows=1 width=24) (actual time=0.104..0.104 rows=1 loops=1) Buffers: shared hit=4 -> Index Scan using qq_p_idx on qq (cost=0.00..82060.60 rows=1000000 width=24) (actual time=0.104..0.104 rows=1 loops=1) Sort Cond: (p <-> '(0,0)'::point) Buffers: shared hit=4 Total runtime: 0.117 ms 4000 times faster ! -------------------------- 24.04.11
  • 31. Knn-search: Examples n=1000,000 k :hit :knn : seq :sortmem --------------------------------------------------- 1 :4 :0.117 :469.150 : 25 10 :17 :0.289 :471.735 : 25 100 :118 :0.872 :468.244 : 32 1000 :1099 :7.107 :473.840 : 127 10000 :10234 :31.629 :525.557 : 1550 100000 :101159:321.182:994.925: 13957 24.04.11
  • 32. Knn-search: Examples n=10,000 K :hit :knn : seq ---------------------------------- 1 :3 :0.117 :6.072 10 :13 :0.247 :5.014 100 :103 :0.295 :6.381 1000 :996 :1.605 :8.670 10000 :9916 :16.487 :14.706 -> knn lose if k=n, n is small 24.04.11
  • 33. Knn-search: Examples ● Real data 2 mln points US, geonames 24.04.11
  • 34. Knn-search: Examples ● Query: find 10 closest points in US with 'mars' in names to the point (5,5) - create composite index: create index pt_fts_idx on geo using gist(point, to_tsvector('english',asciiname)); =# explain (analyze on, buffers on) select asciiname,point, (point <-> '5.0,5.0'::point) as dist from geo where to_tsvector('english', asciiname) @@ to_tsquery('english','mars') order by dist asc limit 10; QUERY PLAN --------------------------------------------------------------------------------------- Limit (cost=0.00..33.55 rows=10 width=35) (actual time=0.452..0.597 rows=10 loops=1) Buffers: shared hit=56 -> Index Scan using pt_fts_idx on geo (cost=0.00..34313.91 rows=10227 width=35) (actual time=0.452..0.592 rows=10 loops=1) Index Cond: (to_tsvector('english'::regconfig, (asciiname)::text) @@ '''mar'''::tsquery) Sort Cond: (point <-> '(5,5)'::point) Buffers: shared hit=56 Total runtime: 0.629 ms (7 rows) 24.04.11
  • 35. Knn-search: Existing solutions knn=# select id, date, event from events order by date <-> '1957-10-04'::date asc limit 10; id | date | event --------+------------+----------------------------------------------------------------- 58137 | 1957-10-04 | U.S.S.R. launches Sputnik I, 1st artificial Earth satellite 58136 | 1957-10-04 | "Leave It to Beaver," debuts on CBS 117062 | 1957-10-04 | Gregory T Linteris, Demarest, New Jersey, astronaut, sk: STS 83 117061 | 1957-10-04 | Christina Smith, born in Miami, Florida, playmate, Mar, 1978 102670 | 1957-10-05 | Larry Saumell, jockey 31456 | 1957-10-03 | Willy Brandt elected mayor of West Berlin 58291 | 1957-10-05 | 12th Ryder Cup: Britain-Ireland, 7 -4 at Lindrick GC, England 58290 | 1957-10-05 | 11th NHL All-Star Game: All-Stars beat Montreal 5-3 at Montreal 58292 | 1957-10-05 | Yugoslav dissident Milovan Djilos sentenced to 7 years 102669 | 1957-10-05 | Jeanne Evert, tennis player, Chris' sister (10 rows) Time: 115.548 ms ● Very inefficient: ● Full table scan, btree index on date won't help. ● Sort full table 24.04.11
  • 36. Knn-search: Existing solutions contrib/btree_gist knn=# select id, date, event from events order by date <-> '1957-10-04'::date asc limit 10; id | date | event --------+------------+----------------------------------------------------------------- 58137 | 1957-10-04 | U.S.S.R. launches Sputnik I, 1st artificial Earth satellite 58136 | 1957-10-04 | "Leave It to Beaver," debuts on CBS 117062 | 1957-10-04 | Gregory T Linteris, Demarest, New Jersey, astronaut, sk: STS 83 117061 | 1957-10-04 | Christina Smith, born in Miami, Florida, playmate, Mar, 1978 102670 | 1957-10-05 | Larry Saumell, jockey 31456 | 1957-10-03 | Willy Brandt elected mayor of West Berlin 58291 | 1957-10-05 | 12th Ryder Cup: Britain-Ireland, 7 -4 at Lindrick GC, England 58290 | 1957-10-05 | 11th NHL All-Star Game: All-Stars beat Montreal 5-3 at Montreal 58292 | 1957-10-05 | Yugoslav dissident Milovan Djilos sentenced to 7 years 102669 | 1957-10-05 | Jeanne Evert, tennis player, Chris' sister (10 rows) Time: 0.590 ms ● Very inefficient: ● 8 index pages read + 10 tuples read ● NO sorting ● About 200 times faster ! 24.04.11
  • 37. Knn-search: Examples ● pg_trgm support – distance = 1 – Similarity knn=# select date, event, ('jeorge ewashington' <-> event ) as dist from events order by dist asc limit 10; date | event | dist ------------+---------------------------------------------------+---------- 1732-02-11 | George Washington | 0.458333 1792-12-05 | George Washington re-elected U.S. pres | 0.674419 1811-02-23 | George Washington Hewitt, composer | 0.675 1753-08-04 | George Washington becomes a master mason | 0.697674 1941-07-19 | Jennifer Dunn, Rep-R-Washington | 0.710526 1945-05-12 | Jayotis Washington, rocker | 0.714286 1817-05-05 | George Washington Julian, MC, Union, died in 1899 | 0.72549 1789-08-25 | Mary Ball Washington, mother of George, dies | 0.729167 1844-01-12 | George Washington Cable, American Novelist | 0.729167 1925-01-31 | George Washington Cable, American Novelist | 0.729167 (10 rows) Time: 187.604 ms 24.04.11
  • 38. Knn-search: Examples ● Corner case for knn-search - all data are on the same distance from point Q ! Q 24.04.11
  • 39. Knn-search: Examples ● Corner case for Best First Strategy - all data are on the same distance from point Q ! create table circle (id serial, p point, s int4); insert into circle (p,s) select point( p.x, p.y), (random()*1000)::int from ( select t.x, sqrt(1- t.x*t.x) as y from ( select random() as x, generate_series(1,1000000) ) as t ) as p; create index circle_p_idx on circle using gist(p); analyze circle; Number of levels: 3 Number of pages: 8266 Number of leaf pages: 8201 24.04.11
  • 40. Knn-search: Examples ● Corner case for knn-search - all data are on the same distance from point Q ! =# explain (analyze on, buffers on) select * from circle order by (p <-> '(0,0)') asc limit 10; Limit (cost=0.00..0.80 rows=10 width=24) (actual time=226.907..226.924 rows=10 loops=1) Buffers: shared hit=8276 -> Index Scan using circle_p_idx on circle (cost=0.00..79976.58 rows=1000000 width=24) (actual time=226.905..226.921 rows=10 loops=1) Sort Cond: (p <-> '(0,0)'::point) Buffers: shared hit=8276 – read all index Total runtime: 230.885 ms ● Still 2 times faster than SEQ (454.331 ms) because of sorting 24.04.11
  • 41. Knn-search: Status Committed to 9.1 24.04.11
  • 42. Knn: History of development ● Начало проекта - Sep 8, 2007 at 7:54 PM date Sat, Sep 8, 2007 at 7:54 PM subject Chat with Sergey V. Karpov 7:36 PM me: я тут knn-search занимаюсь, масса интересного. Все думаю, как в постгресе это поиметь Sergey: а что это такое? 7:37 PM me: k-nearest соседей - супер важная задача найти 5 ближайших точек 7:38 PM Sergey: ближайших к чему? me: к заданной точке 7:39 PM Sergey: в какой системе координат? me: в любой, в n-мерном пространстве. В простом варианте - хотя бы на земле/небе 7:40 PM это нужно для поиска похожих картинок, например. навиный вариант повторять запросы - не катит 24.04.11
  • 43. Knn: History of development ● TODO (http://www.sai.msu.su/~megera/wiki/TODO) начало 2008 года, уже есть понимание что делать ● 25 июля 2009 года – письмо Paul Ramsey (POSTGIS) ● 10 июля 2009 года – контракт Open Planning Project Inc. ● 20 ноября 2009 года – патч KNNGiST v.0.1 (модуль расш) ● Commitfest nightmare ● 22 июля 2010 – KNNGiST (v.0.8), commitfest ● 13 сентября 2010 – KNNGiST (v.0.9) ● 03 декабря 2010 – Tom Lane committed for 9.1 ! ● 21 января 2011 – contrib/btree_gist committed ! 24.04.11
  • 44. Knn: History of development ● Итого: На проект ушло больше 3 лет ! ● Реальное программирование заняло несколько месяцев ● Основные причины: ● Отсутствие поддержки – Слабая информационная связь разработчиков и заказчиков ● Занятость разработчиков ● Усложнение процедуры рассмотрения проектов в сообществе 24.04.11
  • 45. PostgreSQL 9.1 Что планировалось ? 24.04.11
  • 46. Major features PostgreSQL 9.1 Совещание разработчиков в Оттаве, май 2010 24.04.11
  • 48. PostgreSQL 9.1 Что удалось реализовать (основные новые возможности) 24.04.11
  • 49. PostgreSQL 9.1 ● Функциональность: ● Synchronous Replication ● Per-column collation ● Unlogged tables ● Новинки ● SSI (Serializable Snapshot Isolation), статья Michael J. Cahill, Uwe Röhm, Alan D. Fekete. Serializable Isolation for Snapshot Databases. In the Proceedings of the 2008 ACM SIGMOD international conference on management of data, pages 729-738 24.04.11
  • 50. PostgreSQL 9.1: SSI ● До 9.1 ● READ UNCOMMITTED=READ COMMITTED ● REPEATABLE READ=SERIALIZABLE, в текущей транзакции видны все записи зафиксированные до первого запроса или до выполнения команды, изменяющей данные, в рамках этой транзакции. ● 9.1 ● REPEATABLE READ ● SERIALIZABLE, гарантирует последовательное выполнение транзакций - как будто транзакции выполняются последовательно друг за другом. 24.04.11
  • 51. PostgreSQL 9.1 ● Новинки: ● KNNGIST – поиск ближайших соседей, первая реализация для продакшн системы. ● SE/PostgreS - поддержка мандатной политики доступа, совместимой с SE Linux,в виде расширения, который вводит дополнительный слой проверки безопасности на основе использования меток безопасности (security labels). ● wCTE (writable Common Table Expressions) - использование команд, которые изменяют данные (INSERT,UPDATE,DELETE), с оператором WITH. Это позволяет исполнять несколько разных операций в одном запросе. 24.04.11
  • 52. PostgreSQL 9.1: wCTE Получить сводку удаленных постов по пользователю и обновить статистику пользователей. WITH deleted_posts AS ( DELETE FROM posts WHERE created < now() - '6 months'::INTERVAL RETURNING * ), deleted_per_user as ( SELECT user_id, count(*) FROM deleted_posts GROUP BY 1 ) UPDATE counts SET posts_count = posts_count - d.count FROM deleted_per_user d WHERE d.user_id = counts.user_id; 24.04.11
  • 53. PostgreSQL 9.1 ● Расширяемость: ● Extensions - инфраструктура управления расширениями. Команды CREATE/DROP/ALTER EXTENSION позволяют манипулировать расширениями (устанавливать, обновлять, изменять ) как единым целым. ● SQL/MED - поддержка SQL стандарта в части управления внешними (вне базы данных) данными (Management of External Data) для обеспечения прозрачной интеграции со внешними данными. 24.04.11
  • 54. PostgreSQL 9.1: Extensions postgres=# CREATE EXTENSION file_fdw; CREATE EXTENSION postgres=# dx List of installed extensions Name | Version | Schema | Description ----------+---------+------------+------------------------------------------- file_fdw | 1.0 | public | foreign-data wrapper for flat file access plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language (2 rows) postgres=# dx+ file_fdw Objects in extension "file_fdw" Object Description ----------------------------------------- foreign-data wrapper file_fdw function file_fdw_handler() function file_fdw_validator(text[],oid) (3 rows) 24.04.11
  • 55. PostgreSQL 9.1: SQL/MED postgres=# CREATE SERVER file_server FOREIGN DATA WRAPPER file_fdw; postgres=# CREATE FOREIGN TABLE passwd ( username text, pass text, uid int4, gid int4, gecos text, home text, shell text ) SERVER file_server OPTIONS (format 'text', filename '/etc/passwd', delimiter ':', null ''); postgres=# select * from passwd order by uid asc limit 3; username | pass | uid | gid | gecos | home | shell ----------+------+-----+-----+--------+-----------+----------- root | x | 0 | 0 | root | /root | /bin/bash daemon | x | 1 | 1 | daemon | /usr/sbin | /bin/sh bin | x | 2 | 2 | bin | /bin | /bin/sh (3 rows) postgres=# d List of relations Schema | Name | Type | Owner --------+--------+---------------+---------- public | passwd | foreign table | postgres public | test | table | postgres (2 rows) 24.04.11
  • 56. PostgreSQL 9.1 Полный список изменений http://developer.postgresql.org/pgdocs/postgres/release-9-1.html 24.04.11