Sphinx new
- 1. Improving MySQL- based applications performance with Sphinx Maciej Dobrzaoski (Мачей Добжаньски) Percona, Inc.
- 2. INTRODUCTION Who am I? – Consultant at Percona, Inc. – What do I do? • Performance audits • Fix broken systems • Design architectures – Typically work from home
- 3. INTRODUCTION What is Percona, Inc.? – Consulting company – Provides services for MySQL applications – Develops open-source software • Scalability patches for InnoDB • XtraDB storage engine for MySQL • Xtrabackup – free backup solution for InnoDB/XtraDB
- 5. WHAT IS MYSQL? MySQL is... – Open-source relational database management system – Popular enough to assume everyone here knows it
- 7. WHAT IS SPHINX? A standalone full-text search engine – Consists of two major applications • indexer • searchd – More efficient than MySQL FULLTEXT • On larger data sets
- 8. WHAT IS SPHINX? A standalone full-text search engine – Can be easily scaled horizontally • Sphinx indexes can be distributed across many servers • Allows parallel searching • One instance becomes a dispatcher – Forwards queries to other instances – Combines results before sending them back to clients
- 10. WHAT IS SPHINX? Many additional features beyond just full-text search – Indexable attributes for non-FTS filtering • numerical, multi-value and now also text • Example: limit results to rows which have article_score>=2 – Sorting results by an attribute or an expression • Example: @weight+(article_score)*0.1
- 11. WHAT IS SPHINX? Many additional features beyond just full-text search – Grouping results by an attribute • Additional support for timestamp attributes • Returns also row count per group – may be approximate – Calculating expressions • Much faster than in MySQL as per recent benchmarks
- 12. WHAT IS SPHINX? Anything else? – On-line re-indexing – Live index updates – Extensive API available for many programming languages • PHP • Python • Java • many more
- 13. WHAT IS SPHINX? There’s even more! – SphinxQL – MySQL server protocol compatible • Connect with any MySQL client – command line – API call, e.g. mysql_connect() • Run SQL-like queries
- 14. WHAT IS SPHINX? Example use of SphinxQL
- 15. HOW DOES SPHINX WORK WITH MYSQL?
- 16. HOW DOES SPHINX WORK WITH MYSQL? Sphinx is external application; not part of MYSQL – Uses own data files – Needs memory – Has to be queried separately • Sphinx API • SphinxQL • Sphinx Storage Engine for MySQL
- 17. HOW DOES SPHINX WORK WITH MYSQL? Sphinx is external application; not part of MySQL – Updating Sphinx indexes has to be done separately too • Periodic data re-indexing with indexer – Some information may be outdated for a while – Can be optimized through re-indexing the latest changes only • Live index updates from applications – Applications need to write twice to both MySQL and Sphinx – Available only for attributes; full-text updates to come
- 18. HOW DOES MYSQL WORK WITH SPHINX? Example data source for Sphinx index sql_query = SELECT mi.id, mi.movie_id, t.production_year, t.title, mi.info FROM movie_info mi JOIN title t ON t.id = mi.movie_id sql_attr_uint = movie_id sql_attr_uint = production_year • Notice the source can be any valid SQL query – Uses joins to denormalize data for Sphinx • Two integer attributes – movie_id and production_year
- 19. HOW DOES SPHINX WORK WITH MYSQL? Sphinx is not a full database (yet?) – It’s primarily a search engine – It can return values stored as attributes, e.g: movie_id, production_year – …but not any full-text searchable columns – Results from Sphinx can be used to fetch full details from database
- 20. IMPORTANT FACTS TO KNOW ABOUT MYSQL
- 21. IMPORTANT FACTS TO KNOW ABOUT MYSQL Uses B-TREE indexes to improve search performance – Works great for equality operator (=) – …and small range lookups: >, >=, <, <=, IN (list), LIKE • Range size relative to table size, not an absolute value • Large range often turns into plain scan
- 22. IMPORTANT FACTS TO KNOW ABOUT MYSQL MySQL can use any left-most part of an index – INDEX (a, b, c) can fully optimize both: (1) SELECT * FROM T WHERE a=9 (2) SELECT * FROM T WHERE a=9 AND b IN (1,2) AND c=4 …but not any of: (3) SELECT * FROM T WHERE b=7 AND c=1 (4) SELECT * FROM T WHERE a=9 AND c=2 (may still use index for a=9 only) – No good indexes means you may need a new one
- 23. IMPORTANT FACTS TO KNOW ABOUT MYSQL Each index slows down writes to a table – Index is an organized structure, it has to be maintained – There can’t be too many or performance will suffer MySQL can typically use only one index per query – There are rare exceptions – index merge optimizations – Merges are often not good enough – an observation
- 24. IMPORTANT FACTS TO KNOW ABOUT MYSQL These work great in MySQL – Index optimized searching • A query which uses indexes efficiently is fast enough • B-TREE lookups are typically very efficient • FULLTEXT indexes can be the exception – Index optimized sorting and grouping • Rows are read in the proper order
- 25. IMPORTANT FACTS TO KNOW ABOUT MYSQL These can cause problems in MySQL – Full table scans • No index is used • Query reads entire table row by row checking for matches – Large scans related to poor selectivity • An index is used, but it is not selective enough • MySQL has to read a lot of rows and reject many of them
- 26. IMPORTANT FACTS TO KNOW ABOUT MYSQL These can cause problems in MySQL – Search on many combinations of columns in a single table • Each combination may require new index • Can’t have too many indexes in table at the same time – Handling multi-value properties in searches • Keywords, tags • Such queries often can’t be optimized very well
- 27. IMPORTANT FACTS TO KNOW ABOUT MYSQL These can cause problems in MySQL – Sorting or grouping not done through indexes • Requires rewriting rows into temporary storage • At least one additional pass over results to complete • LIMIT does not work until all matches are found and sorted/grouped
- 28. IMPORTANT FACTS TO KNOW ABOUT MYSQL Indexes and data may be cached in memory – key_buffer and filesystem cache for MyISAM tables – innodb_buffer_pool for InnoDB tables – No guarantees what is in RAM • MySQL has no option to lock certain data in buffers
- 29. IMPORTANT FACTS TO KNOW ABOUT MYSQL Full-text support in MySQL – Available through FULLTEXT keys – Only supported by MyISAM engine • MyISAM uses table level locking • May become a showstopper for busy databases – Cannot be used together with any other index • Even index merge will not work
- 30. IMPORTANT FACTS TO KNOW ABOUT SPHINX
- 31. IMPORTANT FACTS TO KNOW ABOUT SPHINX Search remembers no more than max_matches results | total | 1000 | | total_found | 2255 | – Other results are ignored before sending them to client – Saves some CPU and RAM – All results are often unnecessary – Accuracy costs
- 32. IMPORTANT FACTS TO KNOW ABOUT SPHINX
- 33. IMPORTANT FACTS TO KNOW ABOUT SPHINX Grouping is done in fixed memory – Results may be approximate • When number of matches exceeds max_matches – Inaccuracy depends on max_matches setting • The larger the more accurate grouping results • Growing max_matches can reduce performance – Accuracy costs
- 34. IMPORTANT FACTS TO KNOW ABOUT SPHINX MySQL Sphinx (uses SphinxQL) SELECT ..., COUNT(1) _c SELECT * FROM movie_info FROM movies WHERE WHERE MATCH (info) MATCH ('@info "story"') AGAINST ('"story"' GROUP BY movie_id IN BOOLEAN MODE) ORDER BY @count DESC 4 GROUP BY movie_id ORDER BY _c DESC LIMIT 4
- 35. IMPORTANT FACTS TO KNOW ABOUT SPHINX MySQL Sphinx +----------+----------+ +----------+--------+ | movie_id | COUNT(1) | | movie_id | @count | +----------+----------+ +----------+--------+ | 30372 | 15 | | 30372 | 15 | | 855624 | 13 | | 855624 | 13 | | 590071 | 13 | | 143384 | 12 | | 143384 | 12 | | 590071 | 12 | +----------+----------+ +----------+--------+
- 36. IMPORTANT FACTS TO KNOW ABOUT SPHINX Full copy of attributes is always kept in RAM – If attribute storage was set to ‘extern’ – the typical use – Preloaded on start – Never read from disk again once Sphinx is up – Guarantees certain performance – Calculate the storage requirements properly • Sphinx may want to allocate too much memory
- 37. IMPORTANT FACTS TO KNOW ABOUT SPHINX Sphinx stores rows in blocks – 64 rows per block – Meta data contains (min, max) range of every attribute – Allows quick rejection when filtering by attributes • No need to scan every row individually
- 38. MYSQL V SPHINX PERFORMANCE
- 39. FULL-TEXT SEARCH PERFORMANCE USES FULL IMDB DATABASE IMPORTED INTO MYSQL AND INDEXED WITH SPHINX
- 40. FULL-TEXT SEARCH PERFORMANCE MySQL Sphinx (uses SphinxQL) SELECT COUNT(1) SELECT * FROM movie_info FROM movies WHERE WHERE MATCH (info) MATCH ('@info "james AGAINST ('"james bond"' bond"') IN BOOLEAN MODE)
- 41. FULL-TEXT SEARCH PERFORMANCE MySQL Sphinx +----------+ +---------------+-------+ | COUNT(1) | | Variable_name | Value | +----------+ +---------------+-------+ | 2255 | | total | 1000 | +----------+ | total_found | 2255 | 1 row in set (0.13 sec) | time | 0.003 | ...
- 42. SCAN PERFORMANCE USES FULL IMDB DATABASE IMPORTED INTO MYSQL AND INDEXED WITH SPHINX
- 43. SCAN PERFORMANCE MySQL Sphinx (uses SphinxQL) SELECT COUNT(1) SELECT * FROM title FROM titles WHERE WHERE production_year >= 1990 production_year >= 1990 AND AND production_year <= 2000 production_year <= 2000 No index on `production_year`
- 44. SCAN PERFORMANCE MySQL Sphinx +----------+ +---------------+--------+ | COUNT(1) | | Variable_name | Value | +----------+ +---------------+--------+ | 239203 | | total | 1000 | +----------+ | total_found | 239203 | 1 row in set (1.09 sec) | time | 0.051 | ...
- 45. MORE COMPLEX CASE SEARCH BY KEYWORDS USES FULL IMDB DATABASE IMPORTED INTO MYSQL AND INDEXED WITH SPHINX
- 46. SEARCH BY KEYWORDS MySQL Sphinx (uses SphinxQL) SELECT t.id FROM title t SELECT * JOIN movie_keyword mk FROM keywords ON mk.movie_id = t.id WHERE JOIN keyword k ON k.id = mk.keyword_id MATCH ('@keywords WHERE ("beautiful-woman"| k.keyword IN ('beautiful- "women"|"murder")') woman', 'women', 'murder') ORDER BY production_year DESC GROUP BY t.id ORDER BY LIMIT 3 production_year DESC LIMIT 3
- 47. SEARCH BY KEYWORDS MySQL Sphinx +--------+ +--------+ | id | | id | +--------+ +--------+ | 561959 | | 561959 | | 74273 | | 74273 | | 344814 | | 344814 | +--------+ +--------+ 3 rows in set (1.84 sec) time = 0.015
- 48. SEARCH BY KEYWORDS Sphinx returns – Values of the indexed attrubites – Meta information about search and results – No text • Recent version can actually store and return short strings • But only defined as attributes, not full-text searchable
- 49. SEARCH BY KEYWORDS Use that information to fetch full details from MySQL mysql> SELECT t.id, t.title FROM title t WHERE t.id IN(561959, 74273, 344814) +--------+---------------------------------------+ | id | title | +--------+---------------------------------------+ | 74273 | Blue Silence | | 344814 | Marvin: The Life Story of Marvin Gaye | | 561959 | The Red Man's View | +--------+---------------------------------------+
- 50. SEARCH BY KEYWORDS MySQL Sphinx +--------+-------------------+ +--------+-----------------+ | id | title | | id | production_year | +--------+-------------------+ +--------+-----------------+ | 74273 | Blue Silence | | 561959 | 2014 | | 344814 | Marvin: The Li... | | 74273 | 2013 | | 561959 | The Red Man's ... | | 344814 | 2012 | +--------+-------------------+ +--------+-----------------+ Notice MySQL returned rows in different order!
- 51. SEARCH BY KEYWORDS The order in SQL can only be guaranteed with ORDER BY! What is the solution? – Append ORDER BY production_year DESC • applies to only small number of rows, so it’s probably okay or – Remember the order of Sphinx results in application – Restore it after reveiving data from MySQL
- 52. SEARCH BY KEYWORDS What if „keywords” were numerical identifiers? – Create „fake keywords” and index them as text – Convert numbers into strings when building index sql_query = SELECT t.id, GROUP_CONCAT(CONCAT('KEY_', mk.keyword_id)) FROM title t JOIN movie_keyword mk ON t.id = mk.movie_id GROUP BY t.id – Run full-text searches using strings such as "KEY_1234"
- 53. FLEXIBLE SEARCH
- 54. FLEXIBLE SEARCH A data structure describing user profile CREATE TABLE `members` ( `user_id` int(10) unsigned, `user_firstname` varchar(50) unsigned, `user_surname` varchar(50) unsigned, `user_dob` date unsigned, `user_lastvisit` datetime unsigned, `user_datetime` datetime unsigned, `user_bio` unsigned, `user_hasphoto` tinyint(2) unsigned, `user_hasvideo` tinyint(2) unsigned, ...
- 55. FLEXIBLE SEARCH Flexible search typically means – Search conditions may involve any number of columns in any combination – Sorting may be done on one of many columns as well Often impossible to add all necessary indexes in MySQL
- 56. FLEXIBLE SEARCH Many columns may have very low cardinality – Example: user_gender – MySQL would not even consider using index for such column It may be very difficult to make it work fast in MySQL – When tables or traffic are large enough
- 57. FLEXIBLE SEARCH How does Sphinx help? – Scans are optimized – Optimizations apply to all columns – Possibility to use „fake keywords” – Data can be split across several instances • Parallel search • No extra application logic necessary to combine results
- 58. SUMMARY
- 59. SUMMARY Sphinx can be of great help to many MySQL-based apps – Developed to work better where MySQL performs poorly • Text search • Large scans • Filtering on many combinations of columns • Handling multi-value properties
- 60. SUMMARY Sphinx can be of great help to any MySQL-based apps – Comes with features that can actually replace database – Easily scalable – Actively developed – You can sponsor development and have features you need done soon • No need to wait long until some functionality „appears”
- 62. THANK YOU!