MariaDB with SphinxSE
12 likes4,765 views
MariaDB with SphinxSE - SphinxSE allows MariaDB to use Sphinx for full-text searching by acting as a storage engine that communicates with the Sphinx search daemon. - It allows existing applications using MySQL full-text search to port to Sphinx searching more easily. - SphinxSE passes Sphinx queries to searchd and returns the results to MariaDB, supporting many Sphinx query options and optimizations for sorting, filtering, and result slicing.
MariaDB with SphinxSE
- 1. MariaDB with SphinxSE Colin Charles, Monty Program Ab colin@montyprogram.com http://www.montyprogram.com / http://mariadb.org http://bytebot.net/blog / @bytebot on Twitter Sphinx Search Day 2012, Santa Clara, CA, USA 13 April 2012
- 2. whoami • MariaDB guy at Monty Program • Formerly at MySQL AB/Sun Microsystems • Past lives include FESCO (Fedora Project), OpenOffice.org
- 3. MariaDB/MySQL used interchangeably in this talk
- 4. The old days • Download MySQL, including sources • Download SphinxSE for compiling • Download Sphinx to compile with MySQL support • Documented: http://www.howtoforge.com/ sphinx-as-mysql-storage-engine-sphinxse
- 5. Today • Install sphinx from your distribution • Install MariaDB 5.5 from your distribution or from http://mariadb.org/ • Get started!
- 6. Getting started mysql> INSTALL PLUGIN sphinx SONAME 'ha_sphinx.so'; Query OK, 0 rows affected (0.01 sec)
- 8. What is SphinxSE? • SphinxSE is just the storage engine that still depends on the Sphinx daemon • It doesn’t store any data itself • Its just a built-in client to allow MariaDB to talk to Sphinx searchd, run queries, obtain results • Indexing, searching is performed on Sphinx
- 9. Configure sphinx! • /usr/local/sphinx/sphinx.conf • Source (multiple, include mysql, with connection info) • Setup indexer (esp. if its on localhost) - mem_limit, max_iops, max_iosize • Setup searchd (where to listen to, query log, etc.)
- 10. Use case scenarios • Already have an existing application that makes use of full-text-search in MyISAM? Porting should be easier • Have a programming language without a native API for Sphinx? Surely there’s a connector for MariaDB ;-)
- 11. Use case scenarios • Results from Sphinx itself almost always require additional work involving MariaDB • Say to pull out text column that Sphinx index doesn’t store • JOIN with another table (using a different engine)
- 12. An example CREATE TABLE t1 ( id INTEGER UNSIGNED NOT NULL, weight INTEGER NOT NULL, query VARCHAR(3072) NOT NULL, group_id INTEGER, INDEX(query) ) ENGINE=SPHINX CONNECTION="sphinx://localhost:9312/ test"; SELECT * FROM t1 WHERE query='test it;mode=any';
- 13. Sphinx search tables • 1st column: INTEGER UNSIGNED or BIGINT (document ID) • 2nd column: match weight • 3rd column: VARCHAR or TEXT (your query) • Query column needs indexing, no other column needs to be
- 14. What actually happens • SELECT passes a Sphinx query as the query column in the WHERE clause • searchd returns the results • SphinxSE translates and returns the results to MariaDB
- 15. SHOW ENGINE SPHINX STATUS • Per-query & per-word statistics that searchd returns are accessible via SHOW STATUS mysql> SHOW ENGINE SPHINX STATUS; +--------+-------+-------------------------------------------------+ | Type | Name | Status | +--------+-------+-------------------------------------------------+ | SPHINX | stats | total: 25, total found: 25, time: 126, words: 2 | | SPHINX | words | sphinx:591:1256 soft:11076:15945 | +--------+-------+-------------------------------------------------+ 2 rows in set (0.00 sec)
- 16. What queries are supported? • Most of the Sphinx API is exposed to SphinxSE • query, mode, sort, offset, limit, index, minid, maxid, weights, filter, !filter, range, !range, maxmatches, groupby, groupsort, indexweights, comment, select • Sphinx search modes can also be supported via _sph attributes • obtain value of @groupby? use ‘_sph_groupby’
- 17. Efficiency • Allow Sphinx to perform sorting, filtering, and slicing of result set • ... as opposed to using WHERE, ORDER BY, LIMIT clauses on MariaDB • Why? • Sphinx optimises and performs better on these tasks • Less data packed by searchd, and transferred and unpacked by SphinxSE
- 18. JOINs • Perform JOINs on a SphinxSE search table using tables from other engines SELECT content, date_added FROM test.documents docs -> JOIN t1 ON (docs.id=t1.id) -> WHERE query="one document;mode=any"; +-------------------------------------+---------------------+ | content | docdate | +-------------------------------------+---------------------+ | this is my test document number two | 2006-06-17 14:04:28 | | this is my test document number one | 2006-06-17 14:04:28 | +-------------------------------------+---------------------+ 2 rows in set (0.00 sec)
- 19. Why MariaDB? • We keep up to date with Sphinx releases • In MariaDB 5.5.21 we upgraded to 2.0.4, the latest upstream release • MariaDB 5.5.23 is GA and ready for use today
- 20. Why MariaDB II? • Engineering and furthering MySQL happens with MariaDB • Benefit from a better-built in optimizer (that can materialize subqueries), XtraDB, microsecond precision, more statistics, NoSQL-like features (dynamic columns), GIS functionality (which works for geo- distance type searches in Sphinx)
- 21. Warning • If sphinx is itself not setup, SphinxSE will accept doing things like CREATE TABLE • Try doing a SELECT and you’ll see it fail though
- 22. We have extensive documentation • http://kb.askmonty.org/en/sphinx-storage- engine • http://sphinxsearch.com/docs/1.10/ sphinxse-using.html • Introduction to Search with Sphinx by Andrew Aksyonoff (O’Reilly)
- 23. Q&A? email: colin@montyprogram.com http://montyprogram.com/ | http://mariadb.org/ twitter: @bytebot / url: http://bytebot.net/blog/