Practical SPARQL Benchmarking
Download as PPTX, PDF3 likes2,856 views
The document discusses the importance of benchmarking various data storage technologies, emphasizing the need for performance evaluation regardless of the technology used. It reviews several benchmarks, such as the Berlin SPARQL Benchmark and Lehigh University Benchmark, while highlighting a lack of standardized methodologies and transparency in performance measurements. Additionally, it introduces a configurable Java command line tool for benchmarking SPARQL queries, which provides various performance metrics.
Practical SPARQL Benchmarking
- 1. Rob Vesse rvesse@yarcdata.com @RobVesse 1
- 2. Regardless of what technology your solution will be built on (RDBMS, RDF + SPARQL, NoSQL etc) you need to know it performs sufficiently to meet your goals You need to justify option X over option Y Business – Price vs Performance Technical – Does it perform sufficiently? No guarantee that a standard benchmark accurately models your usage 2
- 3. Berlin SPARQL Benchmark (BSBM) Relational style data model Access pattern simulates replacing a traditional RDBMS with a Triple Store Lehigh University Benchmark (LUBM) More typical RDF data model Stores require reasoning to answer the queries correctly SPARQL2Bench (SP2B) Again typical RDF data model Queries designed to be hard – cross products, filters, etc. Generates artificially massive unrealistic results Tests clever optimization and join performance 3
- 4. Often no standardized methodology E.g. only BSBM provides a test harness Lack of transparency as a result If I say I’m 10x faster than you is that really true or did I measure differently? Are the figures you’re comparing with even current? What actually got measured? Time to start responding Time to count all results Something else? Even if you run a benchmark does it actually tell you anything useful? 4
- 5. Java command line tool (and API) for benchmarking Designed to be highly configurable Runs any set of SPARQL queries you can devise against any HTTP based SPARQL endpoint Run single and multi-threaded benchmarks Generates a variety of statistics Methodology Runs some quick sanity tests to check the provided endpoint is up and working Optionally runs W warm up runs prior to actual benchmarking Runs a Query Mix N times Randomizes query order for each run Discards outliers (best and worst runs) Calculates averages, variances and standard deviations over the runs Generates reports as CSV and XML 5
- 6. Response Time Time from when query is issued to when results start being received Runtime Time from when query is issued to all results being received and counted Exact definition may vary according to configuration Queries per Second How many times a given query can be executed per second Query Mixed per Hour How many times a query mix can be executed per hour 6
- 7. 7
- 8. SP2B at 10k, 50k and 250k run with 5 warm-ups and 25 runs All options left as defaults i.e. full result counting Runs for 50k and 250k skipped if store was incapable of performing the run in reasonable time Run on following systems *nix based stores run on late 2011 Mac Book Pro (quad core, 8GB RAM, SSD) Java heap space set to 4GB Windows based stores run on HP Laptop (dual core, 4GB RAM, HDD) Both low powered systems compared to servers Benchmarked Stores Jena TDB 0.9.1 Sesame 2.6.5 (Memory and Native Stores) Bigdata 1.2 (WORM Store) Dydra Virtuoso 6.1.3 (Open Source Edition) dotNetRDF (In-Memory Store) Stardog 0.9.4 (In-Memory and Disk Stores) OWLIM 8
- 9. 9
- 10. 1 0
- 11. 1 1
- 12. Code Release is management Approved Currently undergoing Legal and IP Clearance Should be open sourced shortly under a BSD license Will be available from https://sourceforge.net/p/sparql-query-bm Apologies this isn’t yet available at time of writing Example Results data available from: https://dl.dropbox.com/u/590790/semtech2012.tar.gz 1 2
- 13. 1 3
Editor's Notes
- #2: Introduce MyselfMay want to add a disclaimer here about views/opinions expressed primarily being my personal ones and not those of the company a la DVD extras disclaimers ;-)
- #3: What is says on the slide ;-)
- #4: Describe the benchmarks – shown on slidesDiscuss deficiencies of each benchmarkBSBMRelational – not really showing off the capabilities of a SPARQL engineLUBMNeed for reasoning – implementation thereof can make a huge difference in performanceForward vs Backward Chaining ReasoningSP2BQueries are unrealisticFocuses on optimization
- #5: Self explanatory slide for the most partHighlight that just because the store you are interested in is good/bad at a particular benchmark doesn’t tell you whether the store is good/bad for your use case
- #6: Describe the methodology in detailNote that this is based on an amalgamation of the BSBM style and Revelytix SP2B methodologies
- #7: Key Point is to cover difference between Response Time and RuntimeNote that this stat can give some interesting information about how stores execute queries – almost instant response time but much longer runtime indicates streaming execution. Long response time with small difference to runtime indicates a batch execution.
- #8: Run through a brief demo of the command line tool – make sure to have a running Stardog/Fuseki instance to run against – likely safer to use Fuseki as easier to ensure running and open source so no appearance of bias to a commercial productRun on SP2B 10k – will complete in reasonable time while I’m talking – suggest using a limited number of runs for demo purposes.Show the output data (CSV and XML)Key difference is CSV converts to seconds while XML uses raw nanosecondsXML is better for post processingCSV useful for quick import into Spreadsheet tools
- #9: Discuss the setup for the example results – why the stores were chosen?Ease of availability (open source, runnable on *nix, personal interest etc)Ensure to highlight YMMVDisclaimer – Be sure to state that this is just a arbitrarily selected sample of stores and that performance indicated here may not be representative of the true performance of any store. Most importantly Cray/YarcData is not endorsing any specific store.Again point out the importance of people running their own benchmarks
- #10: Note how as dataset size increases many stores can’t complete within reasonable time on the machines we usedLogarithmic ScaleMake sure to mention that the fact that many stores did not complete on the 50k and 250k sizes doesn’t mean they are defective, merely that with the machine resources available they couldn’t run in a timely fashion. This leads nicely to the point that it is important to benchmark on the hardware you actually intend to use.
- #11: Discuss the variation in average runtime – some stores are way ahead of othersNote that some store’s results are heavily influenced by poor performance on certain queries – see next slideLogarithmic Scale
- #12: Highlight the variation in performance both between stores and queries. Note how certain queries are just fundamentally hard even with clever optimisationIn-Memory trumps disk for relevant stores in most cases