SlideShare a Scribd company logo

Embedding CPython in Solr

Lucidworks (Archived), profile picture
Lucidworks (Archived)

The document discusses the MontySolr project, which connects Python and Java applications seamlessly, especially for the high-energy physics community at CERN. It describes the challenges faced in embedding Solr in Python applications and outlines the technologies and methods adopted to achieve effective integration. The project's open-source nature and the extensive testing demonstrate its robustness and ability to handle large data volumes efficiently.

1 of 85
Downloaded 19 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
MontySolr: Embedding CPython in Solr Roman Chyla, CERN roman.chyla@cern.ch, May 26, 2011 Thursday, May 26, 2011
Why should I care? - Our challenge is to connect Python and Java - Without compromises - We created MontySolr extension - Robust, tested (will be used by our system) - But works for any Python application (eg. Django) - And for any C/C++ app that Python understands! - Open source (GPL v2) - Try it out! - https://github.com/romanchyla/montysolr 2 Thursday, May 26, 2011
Outline ‣ Context - The Challenge - Key components - Available technologies - Our approach - Problems solved - Evaluation - Wrap-up 3 Thursday, May 26, 2011
CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
SPIRES - Stanford Linear Accelerator Center - SLAC - High-Energy Physics Literature Database - Started December 1991 - The first web outside Europe/CERN - The first database on web 5 Thursday, May 26, 2011
SPIRES - Stanford Linear Accelerator Center - SLAC - High-Energy Physics Literature Database - Started December 1991 - The first web outside Europe/CERN - The first database on web 5 Thursday, May 26, 2011
6 Thursday, May 26, 2011
7 Thursday, May 26, 2011
Invenio - Integrated digital library software behind INSPIRE - Used by very large institutional repositories - http://repositories.webometrics.info/toprep_inst.asp - Customizable virtual collections - Flexible management of metadata - 3 000 authors per article - Powerful search engine - Incl. citation map analysis - Written in Python (since 2001) - 290 000 lines of code 8 Thursday, May 26, 2011
Outline - Context ‣ The Challenge - Key components - Available technologies - Our approach - Problems solved - Evaluation - Wrap-up 9 Thursday, May 26, 2011
The Challenge - HEP scientific community - Searches metadata oriented - However fulltexts are changing the situation - And we want to provide even better service - Bigger volumes of data - NLP processing - Semantic search 10 Thursday, May 26, 2011
The Challenge Invenio 11 Thursday, May 26, 2011
The Challenge Query: supersymmetry AND author:ellis Invenio 11 Thursday, May 26, 2011
The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry 11 Thursday, May 26, 2011
The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry IDs: 1;2;3;9.... 11 Thursday, May 26, 2011
The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry IDs: 1;2;3;9.... 11 Thursday, May 26, 2011
The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry IDs: 1;2;3;9.... 11 Thursday, May 26, 2011
The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry IDs: 1;2;3;9.... 11 Thursday, May 26, 2011
The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry 1-6M IDs IDs: 1;2;3;9.... 11 Thursday, May 26, 2011
The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry 1-6M IDs IDs: 1;2;3;9.... 1. only IDs, no score = no ranking 11 Thursday, May 26, 2011
The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry 1-6M IDs IDs: 1;2;3;9.... 2. score merging 1. only IDs, difficult (if no score available) = no ranking 11 Thursday, May 26, 2011
The Challenge 3. push IDs ? Query: supersymmetry AND author:ellis (eg._faceting) Invenio fulltext:supersymmetry 1-6M IDs IDs: 1;2;3;9.... 2. score merging 1. only IDs, difficult (if no score available) = no ranking 11 Thursday, May 26, 2011
What is the “best” solution? - We love Python... - ...and our applications are written in Python... - But what if Solr is the master search engine? - Merge results inside Solr? - Typical size: 1-10 mil. IDs - Expected latency: 1-2 s. - What we want to achieve: - Fast transfer of hits from Invenio to Solr - Leverage the power of both (no compromises) - Developer-friendly integration, simplicity 12 Thursday, May 26, 2011
Outline - Context - The Challenge ‣ Key components - Available technologies - Our approach - Evaluation - Demonstration - Wrap-up 13 Thursday, May 26, 2011
To embed Solr (in Java app) - Your app simulates Java web container? - use EmbeddedSolrServer - It knows nothing about Java servlets? - use DirectConnect class - Maybe we are too lazy? - Embed the web container (in my case Jetty) - Seemed strange (webserver inside webserver) - ... but it worked well 14 Thursday, May 26, 2011
To embed Solr (in Java app) - Your app simulates Java web container? - use EmbeddedSolrServer - It knows nothing about Java servlets? - use DirectConnect class - Maybe we are too lazy? - Embed the web container (in my case Jetty) - Seemed strange (webserver inside webserver) - ... but it worked well 14 Thursday, May 26, 2011
To embed Solr (in Java app) - Your app simulates Java web container? - use EmbeddedSolrServer - It knows nothing about Java servlets? - use DirectConnect class - Maybe we are too lazy? - Embed the web container (in my case Jetty) - Seemed strange (webserver inside webserver) - ... but it worked well 14 Thursday, May 26, 2011
To embed Solr (in Java app) - Your app simulates Java web container? - use EmbeddedSolrServer - It knows nothing about Java servlets? - use DirectConnect class - Maybe we are too lazy? - Embed the web container (in my case Jetty) - Seemed strange (webserver inside webserver) - ... but it worked well 14 Thursday, May 26, 2011
To embed Solr (in Java app) - Your app simulates Java web container? - use EmbeddedSolrServer - It knows nothing about Java servlets? - use DirectConnect class - Maybe we are too lazy? - Embed the web container (in my case Jetty) - Seemed strange (webserver inside webserver) - ... but it worked well 14 Thursday, May 26, 2011
To use Solr in non-Java app - Solr is already usable via HTTP requests, but we need something else here... - Remote objects/calls? - Pyro, execnet, CORBA, SOAP... - or simply pipes? - Access Python from Java? - Jython - JEPP - Access Java from Python? - JPype - JCC 15 Thursday, May 26, 2011
Jython? - Implementation of Python in 100% Java - Both Java and Python code - Truly multithreaded - C modules will not work - but see http://bit.ly/iTRYbb - Slower than CPython 16 Thursday, May 26, 2011
Jython? - Implementation of Python in 100% Java - Both Java and Python code - Truly multithreaded - C modules will not work - but see http://bit.ly/iTRYbb - Slower than CPython 17 Thursday, May 26, 2011
Jython? - Implementation of Python in 100% Java - Both Java and Python code - Truly multithreaded - C modules will not work - but see http://bit.ly/iTRYbb - Slower than CPython 17 Thursday, May 26, 2011
JEPP - Java Embedded Python - Python code runs inside Python interpreter - Embeds CPython interpreter via Java Native Interface (JNI) in Java - http://jepp.sourceforge.net/ - recently updated (27-Jan) - but JCC is more active 18 Thursday, May 26, 2011
JEPP - Java Embedded Python 19 Thursday, May 26, 2011
JCC - Embeds JVM in Python - C++ code generator - C++ object interface wraps a Java library - C++ wrappers conform to Python's C type system - result: complete Python extension module 20 Thursday, May 26, 2011
JCC 21 Thursday, May 26, 2011
JCC 21 Thursday, May 26, 2011
JCC 21 Thursday, May 26, 2011
To use Solr in non-Java app Jython JCC JEPP Python ✓ ✓ CModules Speed ✓ ? No code ✓ ✓ changes Access from ✓ ✓ Python Access from ✓ ... ✓ Java 22 Thursday, May 26, 2011
The first try Invenio Solr JCC 23 Thursday, May 26, 2011
Devil is in details... 24 Thursday, May 26, 2011
GIL - Global Interpreter Lock Unfortunately Python webapp is not like Java... 25 Thursday, May 26, 2011
GIL - Global Interpreter Lock We can have 200 threads, but only 4 will run at time... 26 Thursday, May 26, 2011
GIL - Global Interpreter Lock 27 Thursday, May 26, 2011
Fortunately solution exists - JCC can embed Python inside Java - Special thanks to Andi Vajda! (JCC creator) - We write ‘empty’ classes in Java ... - ... and implement them in Python Python /w Java inside Java /w Python inside 28 Thursday, May 26, 2011
The second try Solr /w Invenio Invenio (backend) frontend XML JCC 29 Thursday, May 26, 2011
Implementing the bridge - Special Java class - With method pythonExtension() - Native method pythonDecRef() - JCC provides its implementation - And number of other native methods - These will be implemented using Python - Like writing JNI Java/C code but without compilation... 30 Thursday, May 26, 2011
MontySolr extension - JCC has great potential, but also added complexity... - So the MontySolr project was born - Modules must be built in shared mode - JCC dynamic library loaded and started from the main thread - Simple mechanism of the Python bridge and message - Configurable handlers on the Python side - Secured dereferencing of the native objects - Threading on the Java side - Multiprocessing on the Python side - Easy ant targets (compilation) ... 31 Thursday, May 26, 2011
Hello World - Java part public class MontySolrBridge extends BasicBridge implements PythonBridge { private long pythonObject; public void pythonExtension(long pythonObject) { this.pythonObject = pythonObject; } public long pythonExtension() { return this.pythonObject; } public void finalize() throws Throwable { pythonDecRef(); } public native void pythonDecRef(); public void sendMessage(PythonMessage message) { PythonVM vm = PythonVM.get(); vm.acquireThreadState(); receive_message(message); vm.releaseThreadState(); } public native void receive_message(PythonMessage message); } 32 Thursday, May 26, 2011
Hello World - Python part from montysolr import MontySolrBridge class SimpleBridge(MontySolrBridge): def __init__(self): super(SimpleBridge, self).__init__() def receive_message(self, message): query = message.getParam(‘query’) message.setResults(‘Hello world!’) print ‘Python received from Java:’, query 33 Thursday, May 26, 2011
Example - running MontySolr - Java side - JRE (32/64 bit) - Standard Solr/Lucene jars - JCC dynamic library - Python side - Python interpreter (32/64 bit) - 4 Python modules (jcc, solr, lucene, montysolr) - In the main thread - First we load JCC - Then start Python interpreter ... - ... load Python handlers 34 Thursday, May 26, 2011
Solr as search service Solr /w Invenio Invenio (backend) frontend XML JCC 35 Thursday, May 26, 2011
Example Solr MyCustom Handler 36 Thursday, May 26, 2011
Example refersto:author:ellis Solr MyCustom Handler 37 Thursday, May 26, 2011
Example - Solr custom handler MontySolrVM.INSTANCE.sendMessage(message); PythonMessage msg = MontySolrVM.INSTANCE .createMessage("perform_search") .setSender("Invenio") .setParam("query","refersto:author:ellis"); MontySolrVM.INSTANCE.sendMessage(msg); Object result = msg.getResults(); if (result != null) { int[] hits = (int[]) message.getResults(); } 38 Thursday, May 26, 2011
Example - JNI connection refersto:author:ellis Solr MyCustom Python Handler Bridge 39 Thursday, May 26, 2011
Example - JNI connection refersto:author:ellis Solr MyCustom Python Invenio Handler Bridge wrappers 40 Thursday, May 26, 2011
Example - Python side # handler is made ‘visible’ at startup SolrpieTarget('Invenio:perform_search', perform_search) # search time - called from Java def perform_search(message): query = message.getParam(“query”) hits = call_real_search(query) # cast Python list into Java array message.setResults(JArray_ints(hits)) 41 Thursday, May 26, 2011
Example refersto:author:ellis Solr Invenio Invenio MyCustom Python Invenio Handler Bridge wrappers Invenio Invenio 42 Thursday, May 26, 2011
Example - Java side again MontySolrVM.INSTANCE.sendMessage(message); PythonMessage msg = MontySolrVM.INSTANCE .createMessage("perform_search") .setSender("Invenio") .setParam("query","refersto:author:ellis"); MontySolrVM.INSTANCE.sendMessage(msg); Object result = msg.getResults(); if (result != null) { int[] hits = (int[]) message.getResults(); } 43 Thursday, May 26, 2011
Solr as search service Solr /w Invenio Apache (backend) webserver XML Invenio Invenio JCC 44 Thursday, May 26, 2011
Outline - Context - The Challenge - Key components - Available technologies - Our approach - Problems solved ‣ Evaluation - Wrap-up 45 Thursday, May 26, 2011
Memory and garbage collection 46 Thursday, May 26, 2011
Comparing speed and load... 47 Thursday, May 26, 2011
The effect of cache 48 Thursday, May 26, 2011
Robust? - Extensive siege tests show very good performance and stability under high load - 100-200 users, complex searches - 50 concurrent users, citation analysis - JCC incurs small overhead - We detected no memory leaks - The same as dbpedia.org - But watch out for errors in C - An error in C module brings down the whole JVM - (errors in pure Python module can be handled) 49 Thursday, May 26, 2011
Easy to develop/maintain? - Added complexity - Java in the toolbox - Need to compile C++ extensions - Python/OS version dependencies - For this we get - Easy integration with Invenio - The best of two applications - A lot of features for free - And we can control Solr from Python! 50 Thursday, May 26, 2011
Outline - Context - The Challenge - Key components - Available technologies - Our approach - Problems solved - Evaluation ‣ Wrap-up 51 Thursday, May 26, 2011
Wrap-up - Our challenge was to connect two different languages/systems - And we wanted to get the best of the two... - So we had to plug Python into Solr - And now our Solr knows citation analysis! - We created MontySolr extension - Robust, tested (will be used by INSPIRE) - Works for any Python application (eg. Django) - And for any C/C++ app that Python understands! - Free software license - Try it out! Help us make it better! - https://github.com/romanchyla/montysolr 52 Thursday, May 26, 2011
Questions? - MontySolr - https://github.com/romanchyla/montysolr - Roman Chyla - Fellow, CERN Scientific Information Service - roman.chyla@cern.ch - @rchyla - https://svnweb.cern.ch/trac/rcarepo Thursday, May 26, 2011
Additional information 54 Thursday, May 26, 2011
Links - Invenio platform - http://invenio-software.org/ - INSPIRE Digital library - http://inspirebeta.net/ - Diagrams of JCC and JEPP - Andreas Schreiber : Mixing Java and Python - http://www.slideshare.net/onyame/mixing-python-and- java - On Jython C Extension API - http://stackoverflow.com/questions/3097466/using- numpy-and-cpython-with-jython - Demo of a running service: - http://insdev01.cern.ch 55 Thursday, May 26, 2011
#1 - How to embed Solr (standard) - solr.client.solrj.embedded.EmbeddedSolrServer 56 Thursday, May 26, 2011
#2 - How to embed Solr (simplified) - solr.servlet.DirectSolrConnection - like previous, but simpler - all the queries are sent as strings, everything is just a string - very flexible and probably suitable for quick integration 57 Thursday, May 26, 2011
#2 - How to embed Solr (simplified) - solr.servlet.DirectSolrConnection - like previous, but simpler - all the queries are sent as strings, everything is just a string - very flexible and probably suitable for quick integration 57 Thursday, May 26, 2011
#3 - Example of a Solr custom handler 58 Thursday, May 26, 2011
#4 - Example Python handler 59 Thursday, May 26, 2011

More Related Content

PPT
On being a citizen
Sarah Rainey
 
PPTX
Excel lesson 1
james savage
 
PPT
Mobile Marketing
DianaGarcia7
 
PDF
Clipping mundo a Vapor 2010 02
Agência DUE
 
PPTX
Kansen KHN voor LOGIS slideshare
Al Sauerfield
 
PPTX
Nort 23 mei
Al Sauerfield
 
PDF
エデルマン・マスターナラティブ
Edelman Japan
 
PPTX
Presentatie social media: Kansen voor Horecaondernemers. De Lier, 6 juni 2011
Al Sauerfield
 
On being a citizen
Sarah Rainey
 
Excel lesson 1
james savage
 
Mobile Marketing
DianaGarcia7
 
Clipping mundo a Vapor 2010 02
Agência DUE
 
Kansen KHN voor LOGIS slideshare
Al Sauerfield
 
Nort 23 mei
Al Sauerfield
 
エデルマン・マスターナラティブ
Edelman Japan
 
Presentatie social media: Kansen voor Horecaondernemers. De Lier, 6 juni 2011
Al Sauerfield
 

Viewers also liked (15)

PDF
2013 エデルマン・トラストバロメーター
Edelman Japan
 
PPT
Career planning final project
weber10
 
PPTX
25 changes
Horselover102
 
PDF
The Zeitgeist Movement
guestc142a8
 
PDF
Clipping 2008 - Vinicola Garibaldi
Agência DUE
 
PPTX
2012 エデルマン・トラストバロメーター
Edelman Japan
 
PPTX
10 Hottest Web-Design Trends | Keyideas Infotech
Keyideas Infotech Private Limited
 
PDF
Testing Experience Magazine Vol.14 June 2011
MIMOS Berhad/Open University Malaysia/Universiti Teknologi Malaysia
 
PDF
2011 エデルマン・ヘルスバロメーター
Edelman Japan
 
PDF
イノベーションとアーンドブランド　～イノベーションを受け入れてもらうために～
Edelman Japan
 
PPTX
2.11 06 sir
Al Sauerfield
 
PDF
Presentation for WOM Marketing Summit 2013 by Edelman Japan
Edelman Japan
 
PPTX
Nóbeles de la paz 1993 a 2013
Salomé Recio Caride
 
PDF
2015 エデルマン・トラストバロメーター
Edelman Japan
 
PPTX
SymetriQ Cloudstorm London March 2010
Johnny Paterson
 
2013 エデルマン・トラストバロメーター
Edelman Japan
 
Career planning final project
weber10
 
25 changes
Horselover102
 
The Zeitgeist Movement
guestc142a8
 
Clipping 2008 - Vinicola Garibaldi
Agência DUE
 
2012 エデルマン・トラストバロメーター
Edelman Japan
 
10 Hottest Web-Design Trends | Keyideas Infotech
Keyideas Infotech Private Limited
 
Testing Experience Magazine Vol.14 June 2011
MIMOS Berhad/Open University Malaysia/Universiti Teknologi Malaysia
 
2011 エデルマン・ヘルスバロメーター
Edelman Japan
 
イノベーションとアーンドブランド　～イノベーションを受け入れてもらうために～
Edelman Japan
 
2.11 06 sir
Al Sauerfield
 
Presentation for WOM Marketing Summit 2013 by Edelman Japan
Edelman Japan
 
Nóbeles de la paz 1993 a 2013
Salomé Recio Caride
 
2015 エデルマン・トラストバロメーター
Edelman Japan
 
SymetriQ Cloudstorm London March 2010
Johnny Paterson
 
Ad

Similar to Embedding CPython in Solr (20)

KEY
Lucene revolutionmontysolr 2011_presentation
lucenerevolution
 
PDF
Information resources in high energy physics
Proyecto CeVALE2
 
PDF
2 francisco del aguila
FESABID
 
PDF
Scientific computing in The Netherlands
Evert Lammerts
 
PDF
ServiceNow Event 15.11.2012 / ITIL for the Enterprise @CERN
René Haeberlin
 
PPTX
Eps edison volta
Edison S.p.A.
 
PDF
Incite Ir Final 7 19 11
Chris O'Neal
 
PDF
Ejournal science edition
ESTHHUB
 
PDF
Understanding the Big Picture of e-Science
Andrew Sallans
 
PDF
ECM Meets the Semantic Web - Nuxeo World 2011
Stefane Fermigier
 
PDF
Cse new graduate_students2011
Masoud Nikravesh
 
DOCX
2009 HEP Science Network Requirements Workshop Final Report
butest
 
PDF
Cern
crowderius
 
PPT
KCL CeRch presentation, enhanced scholarly publications, 27march2012
Nick Jankowski
 
PDF
Toward A Computational Theory of Everything
ijrap
 
PDF
Preservation And Reuse In High Energy Physics Salvatore Mele
DigitalPreservationEurope
 
PDF
[Webinar] Semantic Technologies
Nuxeo
 
PPT
Introduction
sriniefs
 
PPTX
Search, Signals & Sense: An Analytics Fueled Vision
Seth Grimes
 
PDF
2 What Is Science?
Sebastian De Haro
 
Lucene revolutionmontysolr 2011_presentation
lucenerevolution
 
Information resources in high energy physics
Proyecto CeVALE2
 
2 francisco del aguila
FESABID
 
Scientific computing in The Netherlands
Evert Lammerts
 
ServiceNow Event 15.11.2012 / ITIL for the Enterprise @CERN
René Haeberlin
 
Eps edison volta
Edison S.p.A.
 
Incite Ir Final 7 19 11
Chris O'Neal
 
Ejournal science edition
ESTHHUB
 
Understanding the Big Picture of e-Science
Andrew Sallans
 
ECM Meets the Semantic Web - Nuxeo World 2011
Stefane Fermigier
 
Cse new graduate_students2011
Masoud Nikravesh
 
2009 HEP Science Network Requirements Workshop Final Report
butest
 
Cern
crowderius
 
KCL CeRch presentation, enhanced scholarly publications, 27march2012
Nick Jankowski
 
Toward A Computational Theory of Everything
ijrap
 
Preservation And Reuse In High Energy Physics Salvatore Mele
DigitalPreservationEurope
 
[Webinar] Semantic Technologies
Nuxeo
 
Introduction
sriniefs
 
Search, Signals & Sense: An Analytics Fueled Vision
Seth Grimes
 
2 What Is Science?
Sebastian De Haro
 
Ad

More from Lucidworks (Archived) (20)

PDF
Integrating Hadoop & Solr
Lucidworks (Archived)
 
PDF
The Data-Driven Paradigm
Lucidworks (Archived)
 
PDF
Downtown SF Lucene/Solr Meetup - September 17: Thoth: Real-time Solr Monitori...
Lucidworks (Archived)
 
PDF
SFBay Area Solr Meetup - July 15th: Integrating Hadoop and Solr
Lucidworks (Archived)
 
PPTX
SFBay Area Solr Meetup - June 18th: Box + Solr = Content Search for Business
Lucidworks (Archived)
 
PPTX
SFBay Area Solr Meetup - June 18th: Benchmarking Solr Performance
Lucidworks (Archived)
 
PPTX
Chicago Solr Meetup - June 10th: This Ain't Your Parents' Search Engine
Lucidworks (Archived)
 
PPTX
Chicago Solr Meetup - June 10th: Exploring Hadoop with Search
Lucidworks (Archived)
 
PPTX
What's new in solr june 2014
Lucidworks (Archived)
 
PPTX
Minneapolis Solr Meetup - May 28, 2014: eCommerce Search with Apache Solr
Lucidworks (Archived)
 
PPTX
Minneapolis Solr Meetup - May 28, 2014: Target.com Search
Lucidworks (Archived)
 
PPTX
Exploration of multidimensional biomedical data in pub chem, Presented by Lia...
Lucidworks (Archived)
 
PDF
Unstructured Or: How I Learned to Stop Worrying and Love the xml, Presented...
Lucidworks (Archived)
 
PDF
Building a Lightweight Discovery Interface for Chinese Patents, Presented by ...
Lucidworks (Archived)
 
PDF
Big Data Challenges, Presented by Wes Caldwell at SolrExchage DC
Lucidworks (Archived)
 
PPTX
What's New in Lucene/Solr Presented by Grant Ingersoll at SolrExchage DC
Lucidworks (Archived)
 
PPTX
Solr At AOL, Presented by Sean Timm at SolrExchage DC
Lucidworks (Archived)
 
PPTX
Intro to Solr Cloud, Presented by Tim Potter at SolrExchage DC
Lucidworks (Archived)
 
PPTX
Test Driven Relevancy, Presented by Doug Turnbull at SolrExchage DC
Lucidworks (Archived)
 
PPTX
Building a data driven search application with LucidWorks SiLK
Lucidworks (Archived)
 
Integrating Hadoop & Solr
Lucidworks (Archived)
 
The Data-Driven Paradigm
Lucidworks (Archived)
 
Downtown SF Lucene/Solr Meetup - September 17: Thoth: Real-time Solr Monitori...
Lucidworks (Archived)
 
SFBay Area Solr Meetup - July 15th: Integrating Hadoop and Solr
Lucidworks (Archived)
 
SFBay Area Solr Meetup - June 18th: Box + Solr = Content Search for Business
Lucidworks (Archived)
 
SFBay Area Solr Meetup - June 18th: Benchmarking Solr Performance
Lucidworks (Archived)
 
Chicago Solr Meetup - June 10th: This Ain't Your Parents' Search Engine
Lucidworks (Archived)
 
Chicago Solr Meetup - June 10th: Exploring Hadoop with Search
Lucidworks (Archived)
 
What's new in solr june 2014
Lucidworks (Archived)
 
Minneapolis Solr Meetup - May 28, 2014: eCommerce Search with Apache Solr
Lucidworks (Archived)
 
Minneapolis Solr Meetup - May 28, 2014: Target.com Search
Lucidworks (Archived)
 
Exploration of multidimensional biomedical data in pub chem, Presented by Lia...
Lucidworks (Archived)
 
Unstructured Or: How I Learned to Stop Worrying and Love the xml, Presented...
Lucidworks (Archived)
 
Building a Lightweight Discovery Interface for Chinese Patents, Presented by ...
Lucidworks (Archived)
 
Big Data Challenges, Presented by Wes Caldwell at SolrExchage DC
Lucidworks (Archived)
 
What's New in Lucene/Solr Presented by Grant Ingersoll at SolrExchage DC
Lucidworks (Archived)
 
Solr At AOL, Presented by Sean Timm at SolrExchage DC
Lucidworks (Archived)
 
Intro to Solr Cloud, Presented by Tim Potter at SolrExchage DC
Lucidworks (Archived)
 
Test Driven Relevancy, Presented by Doug Turnbull at SolrExchage DC
Lucidworks (Archived)
 
Building a data driven search application with LucidWorks SiLK
Lucidworks (Archived)
 

Embedding CPython in Solr

  • 1. MontySolr: Embedding CPython in Solr Roman Chyla, CERN roman.chyla@cern.ch, May 26, 2011 Thursday, May 26, 2011
  • 2. Why should I care? - Our challenge is to connect Python and Java - Without compromises - We created MontySolr extension - Robust, tested (will be used by our system) - But works for any Python application (eg. Django) - And for any C/C++ app that Python understands! - Open source (GPL v2) - Try it out! - https://github.com/romanchyla/montysolr 2 Thursday, May 26, 2011
  • 3. Outline ‣ Context - The Challenge - Key components - Available technologies - Our approach - Problems solved - Evaluation - Wrap-up 3 Thursday, May 26, 2011
  • 4. CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
  • 5. CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
  • 6. CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
  • 7. CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
  • 8. CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
  • 9. CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
  • 10. CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
  • 11. CERN - European Organization for Nuclear Research - Switzerland, Geneva - The largest laboratory for High Energy Physics - Home to the Large Hadron Collider - 40-50K HEP scientists worldwide 4 Thursday, May 26, 2011
  • 12. SPIRES - Stanford Linear Accelerator Center - SLAC - High-Energy Physics Literature Database - Started December 1991 - The first web outside Europe/CERN - The first database on web 5 Thursday, May 26, 2011
  • 13. SPIRES - Stanford Linear Accelerator Center - SLAC - High-Energy Physics Literature Database - Started December 1991 - The first web outside Europe/CERN - The first database on web 5 Thursday, May 26, 2011
  • 16. Invenio - Integrated digital library software behind INSPIRE - Used by very large institutional repositories - http://repositories.webometrics.info/toprep_inst.asp - Customizable virtual collections - Flexible management of metadata - 3 000 authors per article - Powerful search engine - Incl. citation map analysis - Written in Python (since 2001) - 290 000 lines of code 8 Thursday, May 26, 2011
  • 17. Outline - Context ‣ The Challenge - Key components - Available technologies - Our approach - Problems solved - Evaluation - Wrap-up 9 Thursday, May 26, 2011
  • 18. The Challenge - HEP scientific community - Searches metadata oriented - However fulltexts are changing the situation - And we want to provide even better service - Bigger volumes of data - NLP processing - Semantic search 10 Thursday, May 26, 2011
  • 19. The Challenge Invenio 11 Thursday, May 26, 2011
  • 20. The Challenge Query: supersymmetry AND author:ellis Invenio 11 Thursday, May 26, 2011
  • 21. The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry 11 Thursday, May 26, 2011
  • 22. The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry IDs: 1;2;3;9.... 11 Thursday, May 26, 2011
  • 23. The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry IDs: 1;2;3;9.... 11 Thursday, May 26, 2011
  • 24. The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry IDs: 1;2;3;9.... 11 Thursday, May 26, 2011
  • 25. The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry IDs: 1;2;3;9.... 11 Thursday, May 26, 2011
  • 26. The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry 1-6M IDs IDs: 1;2;3;9.... 11 Thursday, May 26, 2011
  • 27. The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry 1-6M IDs IDs: 1;2;3;9.... 1. only IDs, no score = no ranking 11 Thursday, May 26, 2011
  • 28. The Challenge Query: supersymmetry AND author:ellis Invenio fulltext:supersymmetry 1-6M IDs IDs: 1;2;3;9.... 2. score merging 1. only IDs, difficult (if no score available) = no ranking 11 Thursday, May 26, 2011
  • 29. The Challenge 3. push IDs ? Query: supersymmetry AND author:ellis (eg._faceting) Invenio fulltext:supersymmetry 1-6M IDs IDs: 1;2;3;9.... 2. score merging 1. only IDs, difficult (if no score available) = no ranking 11 Thursday, May 26, 2011
  • 30. What is the “best” solution? - We love Python... - ...and our applications are written in Python... - But what if Solr is the master search engine? - Merge results inside Solr? - Typical size: 1-10 mil. IDs - Expected latency: 1-2 s. - What we want to achieve: - Fast transfer of hits from Invenio to Solr - Leverage the power of both (no compromises) - Developer-friendly integration, simplicity 12 Thursday, May 26, 2011
  • 31. Outline - Context - The Challenge ‣ Key components - Available technologies - Our approach - Evaluation - Demonstration - Wrap-up 13 Thursday, May 26, 2011
  • 32. To embed Solr (in Java app) - Your app simulates Java web container? - use EmbeddedSolrServer - It knows nothing about Java servlets? - use DirectConnect class - Maybe we are too lazy? - Embed the web container (in my case Jetty) - Seemed strange (webserver inside webserver) - ... but it worked well 14 Thursday, May 26, 2011
  • 33. To embed Solr (in Java app) - Your app simulates Java web container? - use EmbeddedSolrServer - It knows nothing about Java servlets? - use DirectConnect class - Maybe we are too lazy? - Embed the web container (in my case Jetty) - Seemed strange (webserver inside webserver) - ... but it worked well 14 Thursday, May 26, 2011
  • 34. To embed Solr (in Java app) - Your app simulates Java web container? - use EmbeddedSolrServer - It knows nothing about Java servlets? - use DirectConnect class - Maybe we are too lazy? - Embed the web container (in my case Jetty) - Seemed strange (webserver inside webserver) - ... but it worked well 14 Thursday, May 26, 2011
  • 35. To embed Solr (in Java app) - Your app simulates Java web container? - use EmbeddedSolrServer - It knows nothing about Java servlets? - use DirectConnect class - Maybe we are too lazy? - Embed the web container (in my case Jetty) - Seemed strange (webserver inside webserver) - ... but it worked well 14 Thursday, May 26, 2011
  • 36. To embed Solr (in Java app) - Your app simulates Java web container? - use EmbeddedSolrServer - It knows nothing about Java servlets? - use DirectConnect class - Maybe we are too lazy? - Embed the web container (in my case Jetty) - Seemed strange (webserver inside webserver) - ... but it worked well 14 Thursday, May 26, 2011
  • 37. To use Solr in non-Java app - Solr is already usable via HTTP requests, but we need something else here... - Remote objects/calls? - Pyro, execnet, CORBA, SOAP... - or simply pipes? - Access Python from Java? - Jython - JEPP - Access Java from Python? - JPype - JCC 15 Thursday, May 26, 2011
  • 38. Jython? - Implementation of Python in 100% Java - Both Java and Python code - Truly multithreaded - C modules will not work - but see http://bit.ly/iTRYbb - Slower than CPython 16 Thursday, May 26, 2011
  • 39. Jython? - Implementation of Python in 100% Java - Both Java and Python code - Truly multithreaded - C modules will not work - but see http://bit.ly/iTRYbb - Slower than CPython 17 Thursday, May 26, 2011
  • 40. Jython? - Implementation of Python in 100% Java - Both Java and Python code - Truly multithreaded - C modules will not work - but see http://bit.ly/iTRYbb - Slower than CPython 17 Thursday, May 26, 2011
  • 41. JEPP - Java Embedded Python - Python code runs inside Python interpreter - Embeds CPython interpreter via Java Native Interface (JNI) in Java - http://jepp.sourceforge.net/ - recently updated (27-Jan) - but JCC is more active 18 Thursday, May 26, 2011
  • 42. JEPP - Java Embedded Python 19 Thursday, May 26, 2011
  • 43. JCC - Embeds JVM in Python - C++ code generator - C++ object interface wraps a Java library - C++ wrappers conform to Python's C type system - result: complete Python extension module 20 Thursday, May 26, 2011
  • 44. JCC 21 Thursday, May 26, 2011
  • 45. JCC 21 Thursday, May 26, 2011
  • 46. JCC 21 Thursday, May 26, 2011
  • 47. To use Solr in non-Java app Jython JCC JEPP Python ✓ ✓ CModules Speed ✓ ? No code ✓ ✓ changes Access from ✓ ✓ Python Access from ✓ ... ✓ Java 22 Thursday, May 26, 2011
  • 48. The first try Invenio Solr JCC 23 Thursday, May 26, 2011
  • 49. Devil is in details... 24 Thursday, May 26, 2011
  • 50. GIL - Global Interpreter Lock Unfortunately Python webapp is not like Java... 25 Thursday, May 26, 2011
  • 51. GIL - Global Interpreter Lock We can have 200 threads, but only 4 will run at time... 26 Thursday, May 26, 2011
  • 52. GIL - Global Interpreter Lock 27 Thursday, May 26, 2011
  • 53. Fortunately solution exists - JCC can embed Python inside Java - Special thanks to Andi Vajda! (JCC creator) - We write ‘empty’ classes in Java ... - ... and implement them in Python Python /w Java inside Java /w Python inside 28 Thursday, May 26, 2011
  • 54. The second try Solr /w Invenio Invenio (backend) frontend XML JCC 29 Thursday, May 26, 2011
  • 55. Implementing the bridge - Special Java class - With method pythonExtension() - Native method pythonDecRef() - JCC provides its implementation - And number of other native methods - These will be implemented using Python - Like writing JNI Java/C code but without compilation... 30 Thursday, May 26, 2011
  • 56. MontySolr extension - JCC has great potential, but also added complexity... - So the MontySolr project was born - Modules must be built in shared mode - JCC dynamic library loaded and started from the main thread - Simple mechanism of the Python bridge and message - Configurable handlers on the Python side - Secured dereferencing of the native objects - Threading on the Java side - Multiprocessing on the Python side - Easy ant targets (compilation) ... 31 Thursday, May 26, 2011
  • 57. Hello World - Java part public class MontySolrBridge extends BasicBridge implements PythonBridge { private long pythonObject; public void pythonExtension(long pythonObject) { this.pythonObject = pythonObject; } public long pythonExtension() { return this.pythonObject; } public void finalize() throws Throwable { pythonDecRef(); } public native void pythonDecRef(); public void sendMessage(PythonMessage message) { PythonVM vm = PythonVM.get(); vm.acquireThreadState(); receive_message(message); vm.releaseThreadState(); } public native void receive_message(PythonMessage message); } 32 Thursday, May 26, 2011
  • 58. Hello World - Python part from montysolr import MontySolrBridge class SimpleBridge(MontySolrBridge): def __init__(self): super(SimpleBridge, self).__init__() def receive_message(self, message): query = message.getParam(‘query’) message.setResults(‘Hello world!’) print ‘Python received from Java:’, query 33 Thursday, May 26, 2011
  • 59. Example - running MontySolr - Java side - JRE (32/64 bit) - Standard Solr/Lucene jars - JCC dynamic library - Python side - Python interpreter (32/64 bit) - 4 Python modules (jcc, solr, lucene, montysolr) - In the main thread - First we load JCC - Then start Python interpreter ... - ... load Python handlers 34 Thursday, May 26, 2011
  • 60. Solr as search service Solr /w Invenio Invenio (backend) frontend XML JCC 35 Thursday, May 26, 2011
  • 61. Example Solr MyCustom Handler 36 Thursday, May 26, 2011
  • 62. Example refersto:author:ellis Solr MyCustom Handler 37 Thursday, May 26, 2011
  • 63. Example - Solr custom handler MontySolrVM.INSTANCE.sendMessage(message); PythonMessage msg = MontySolrVM.INSTANCE .createMessage("perform_search") .setSender("Invenio") .setParam("query","refersto:author:ellis"); MontySolrVM.INSTANCE.sendMessage(msg); Object result = msg.getResults(); if (result != null) { int[] hits = (int[]) message.getResults(); } 38 Thursday, May 26, 2011
  • 64. Example - JNI connection refersto:author:ellis Solr MyCustom Python Handler Bridge 39 Thursday, May 26, 2011
  • 65. Example - JNI connection refersto:author:ellis Solr MyCustom Python Invenio Handler Bridge wrappers 40 Thursday, May 26, 2011
  • 66. Example - Python side # handler is made ‘visible’ at startup SolrpieTarget('Invenio:perform_search', perform_search) # search time - called from Java def perform_search(message): query = message.getParam(“query”) hits = call_real_search(query) # cast Python list into Java array message.setResults(JArray_ints(hits)) 41 Thursday, May 26, 2011
  • 67. Example refersto:author:ellis Solr Invenio Invenio MyCustom Python Invenio Handler Bridge wrappers Invenio Invenio 42 Thursday, May 26, 2011
  • 68. Example - Java side again MontySolrVM.INSTANCE.sendMessage(message); PythonMessage msg = MontySolrVM.INSTANCE .createMessage("perform_search") .setSender("Invenio") .setParam("query","refersto:author:ellis"); MontySolrVM.INSTANCE.sendMessage(msg); Object result = msg.getResults(); if (result != null) { int[] hits = (int[]) message.getResults(); } 43 Thursday, May 26, 2011
  • 69. Solr as search service Solr /w Invenio Apache (backend) webserver XML Invenio Invenio JCC 44 Thursday, May 26, 2011
  • 70. Outline - Context - The Challenge - Key components - Available technologies - Our approach - Problems solved ‣ Evaluation - Wrap-up 45 Thursday, May 26, 2011
  • 71. Memory and garbage collection 46 Thursday, May 26, 2011
  • 72. Comparing speed and load... 47 Thursday, May 26, 2011
  • 73. The effect of cache 48 Thursday, May 26, 2011
  • 74. Robust? - Extensive siege tests show very good performance and stability under high load - 100-200 users, complex searches - 50 concurrent users, citation analysis - JCC incurs small overhead - We detected no memory leaks - The same as dbpedia.org - But watch out for errors in C - An error in C module brings down the whole JVM - (errors in pure Python module can be handled) 49 Thursday, May 26, 2011
  • 75. Easy to develop/maintain? - Added complexity - Java in the toolbox - Need to compile C++ extensions - Python/OS version dependencies - For this we get - Easy integration with Invenio - The best of two applications - A lot of features for free - And we can control Solr from Python! 50 Thursday, May 26, 2011
  • 76. Outline - Context - The Challenge - Key components - Available technologies - Our approach - Problems solved - Evaluation ‣ Wrap-up 51 Thursday, May 26, 2011
  • 77. Wrap-up - Our challenge was to connect two different languages/systems - And we wanted to get the best of the two... - So we had to plug Python into Solr - And now our Solr knows citation analysis! - We created MontySolr extension - Robust, tested (will be used by INSPIRE) - Works for any Python application (eg. Django) - And for any C/C++ app that Python understands! - Free software license - Try it out! Help us make it better! - https://github.com/romanchyla/montysolr 52 Thursday, May 26, 2011
  • 78. Questions? - MontySolr - https://github.com/romanchyla/montysolr - Roman Chyla - Fellow, CERN Scientific Information Service - roman.chyla@cern.ch - @rchyla - https://svnweb.cern.ch/trac/rcarepo Thursday, May 26, 2011
  • 79. Additional information 54 Thursday, May 26, 2011
  • 80. Links - Invenio platform - http://invenio-software.org/ - INSPIRE Digital library - http://inspirebeta.net/ - Diagrams of JCC and JEPP - Andreas Schreiber : Mixing Java and Python - http://www.slideshare.net/onyame/mixing-python-and- java - On Jython C Extension API - http://stackoverflow.com/questions/3097466/using- numpy-and-cpython-with-jython - Demo of a running service: - http://insdev01.cern.ch 55 Thursday, May 26, 2011
  • 81. #1 - How to embed Solr (standard) - solr.client.solrj.embedded.EmbeddedSolrServer 56 Thursday, May 26, 2011
  • 82. #2 - How to embed Solr (simplified) - solr.servlet.DirectSolrConnection - like previous, but simpler - all the queries are sent as strings, everything is just a string - very flexible and probably suitable for quick integration 57 Thursday, May 26, 2011
  • 83. #2 - How to embed Solr (simplified) - solr.servlet.DirectSolrConnection - like previous, but simpler - all the queries are sent as strings, everything is just a string - very flexible and probably suitable for quick integration 57 Thursday, May 26, 2011
  • 84. #3 - Example of a Solr custom handler 58 Thursday, May 26, 2011
  • 85. #4 - Example Python handler 59 Thursday, May 26, 2011