Stratio's Cassandra Lucene index: Geospatial Use Cases (Andrés de la Peña & Jonathan Nappee, Nephila) | C* Summit 2016

Editor's Notes

• #2: Hello everyone, my name is Andrés de la Peña, from Stratio, and this is Jonathan Nappée, from Nephila Capital. Today, we are going to talk about how to indéx Geospatial data in Cassandra using Stratio's pluggable Lucene's secondary índex, and we will show some examples about how to apply these features to several Nephila's use cases.
• #3: To begin with, I'd like to introduce Stratio. Stratio is a big data company founded in 2013, that currently has more than 200 employees. Our technical team is currently located in Madrid but we also have offices in San Francisco and Bogotá. We focus on óffering a big data platform based on the Spark ecosystem, and we are one of the existing certified spark distributions.
• #4: The presentation has three main points: At first, a quick overview of Stratio's Lucene secondary indexes. Then, we will review the geospatial search features of the plugin. And, finally, Jonathan will show how these geospatial features are applied to three Nephila's business use cases
• #5: Stratio's Lucene index is an open source implementation of Cassandra's secondary indexes based on Lucene. It was first created in 2014 as a fork of Cassandra, and it became a plugin for Apache Cassandra during last year. It extends Cassandra index functionality to provide near real time search such as ElasticSearch or Solr, including full text search capabilities and free multidimensional, geospatial and bitemporal search.
• #6: Rather than building our own index structures, we chose using Apache Lucene as the underlying technology for several reasons: - It is a proven stable and fast índexing solution. - It has a lot of interesting features, such as boolean queries, range queries or relevance search. - Solr and ElasticSearch are successful examples of distributed search éngines built on top of Lucene. - We also like that Lucene is just a small library that can be embedded directly in Cassandra, and not an external service. - Finally, it is an Apache project, like Cassandra, fully open source and with a large user community.
• #7: Here we can see how the integration between Cassandra and Lucene works. There is a Lucene índex embedded in each Cassandra node, so each node indexes its own data. This way, Lucene doesn't have anything to do with distribution and replication, which are responsibility of Cassandra. The peer-to-peer architecture is preserved, so each node is able to coordinate any query. So, no master nodes or external coordinators are required. A cool feature of the Lucene index is that it allows to paginate over rows sorted with a different order than the defined by the partitioner. Sorted pagination is possible thanks to a custom CQL query handler able to intercept and rewrite the Cassandra internal read commands. If we are performing one of these top-k queries, then all the involved nodes will be queried in a parallel fashion. Otherwise, nodes will be sequentially scanned until we find the requested results.
• #8: Now we are going to see how Lucene indexes are created using the Cassandra Query Language. Let's say that we have a table containing tweets. We store the tweet ID, its creation date, the message body, the user ID and the user name. Then, we create the índex in CQL specifying the Stratio índex class and the índexing properties. We specify an índex readers refresh of one second, and which columns are going to be índexed and how. The creation date will be índexed as a date, using a pattern composed by year, month, and day, defining a precision of days The message field will be treated as English tokenized text And the user ID and the hashtags will be indexed as untokenized text.
• #9: Here we have an example showing how to do searches using the Lucene índex. It includes filtering, negative filtering, sorting and routing. We embed the Lucene index JSON syntax inside Cassandra Query Language using the recently created clause for custom expressions. The JSON Lucene expression specifies that we are searching for all the tweets containing the phrase 'cassandra is cool'. We also require that the matched tweets must not be labeled with the hashtag 'cassandra'. Additionally, results should be returned sorted by descending creation date. Also, we add some CQL regular clauses specifying that we are only interested in tweets created by a specific user during this year. 'user' is the partition key of the table, so the search will be routed to a single node, avoiding the performance problems of unrestricted secondary indexes queries. So, with this example we can see how the Lucene índex allows to perform quite complex searches.
• #10: The index is distributed together with a fluent Java query builder that allows to programmatically build index related JSON queries. The built Lucene index clauses are managed as plain strings, so the query builder should be compatible with most JVM-based Cassandra clients, including the popular DataStax Java driver. This is useful because it can be easily integrated in your existing programs. In this example, we show how to use the query builder to write the query that we saw in the previous slide. The produced JSON string can be easily used as a clause of the query builder provided by the DataStax Java driver.
• #11: A very important feature of our indexes is that they can be combined with MapReduce frameworks, especially with Spark. The usage of Lucene predicates with Spark allows to filter the rows at the data database level. This way, we can retrieve from Cassandra only the information that we need. It avoids the unnecessary reads produced by the usual systematic full table scan. And, it can reduce the amount of data to be processed, speeding up the jobs. As you know, in this kind of deployments there is a Spark Worker running in each Cassandra node. This is done in order to parallelize jobs preserving data locality. Since each node indexes its own data, the locality is guaranteed when using Lucene indexes.
• #12: Now we are going to talk about the spatial search capabilities that the Lucene plugin adds to Cassandra. These features are based on the Lucene speishal module and on the Java Topology Suite. A small set of these spatial capabilities was presented during our talk in the last Cassandra Summit. During this year, the number of spatial features has grown significantly, and Nephila has had an important role in this.
• #13: Here we have an example to show how to indéx latitude-longitude points stored in Cassandra using CQL. Imagine an application where we want to find restaurants around you. In this example, the table (on the left) contains the restaurant name, and its location. There isn`t a native point type in Cassandra, so we will use two numeric columns, latitude and longitude, to represent the location. Also a tuple or an UDT could have been used. Then, we create the índex using the statement on the right. In order to indéx the location, we will add a 'geo_point' mapper named 'location'. With this mapping we must specify which columns, in the indexed table, store the latitude and the longitude. We may combine the geo point mapper with any other non-geospatial mapper. For example, we indéx the 'stars' column as an integer.
• #14: Now that the user locations have been índexed, we can start searching for geospeishal data. The simplest type of query that we have is Bounding box. In this example, we search for restaurants placed inside the visible screen of an hypothetical mobile application. To define the bounding box, we specify, in the query, the minimum and maximum latitude and longitude values. This way, we will collect all the restaurants within the specified coordinates
• #15: Another possible query is to search for restaurants placed inside a specific distance range from a fixed point. For this, we must specify the latitude and the longitude of the reference point, and the desired distance range. Max distance is mandatory, whereas min distance is optional. Along with the distance value we can specify a distance unit. In our example, we search for restaurants located at least one hundred meters away but no more than two kilometers away from our position.
• #16: Additionally, it is also possible to sort the results of any search by their distance to a specific location. In the example we request the restaurants closest to the user's location. The 'reverse' attribute controls whether the order should be ascending or descending, that is, if we are going to retrieve the closest or the farthest locations. Finally, we use the CQL limit clause to select only the ten closest restaurants. Although pure sorting queries are perfectly possible, it is usually a good idea to combine sorting with any other filter. This way, we will reduce the number of matched rows, and so the number of locations to be sorted.
• #17: One of the most exciting Lucene features, that we have recently added, is the ability to index complex geographical shapes, and not only latitude-longitude points. The shapes should be stored in Cassandra in text columns in the WKT format. WKT is a popular markup language able to represent points, line strings, polygons and their muliparts. The indexing is based in the JTS library and its integration with Spatial4j. Although WKT is the only currently supported format, we plan to add support for other popular formats such as GeoJSON. In the example we can see a table storing places, where the text column 'shape' stores a WKT geographical shape. We create a Lucene index that maps this column as a geographical shape, specifying the maximum number of levels in the geohash search prefix tree. It is also possible to specify a sequence of geometric transformations to be sequentially applied to the shape before indexing it. Now we will see some examples to demonstrate the utility of these transformations.
• #18: One of the available index-time transformations is calculating the centroid of the indexed shape. In these example the indexed table stores polygons but we are only interested in indexing the center of the shapes.
• #19: Another very useful transformation is indexing a buffer around the initial shape. In the example we are applying a 'buffer' transformation to index the region which is 50km around the stored shape. This transformation could be used, for example, for storing the coverage area of a set of antennas given their lat-lon locations. It could also be used for storing the area around roads or borders defined as line strings.
• #20: Index size and performance greatly depends on the complexity of the indexed shapes. Shapes with a lot of points and precision decimals will produce many terms in the search tree. This increases the size of the index and reduces performance. So, if your use case allows it, it could be interesting to use transformations to simplify the indexed shapes. Both centroid and bounding box are typical transformations to do this. Convex hull can also be an interesting, more accurate, precision reducer. In this example we show how convex hull transformation is used to reduce a complex shape with more than two thousand points, to a simple polygon with only eight points, dramatically increasing both indexing and searching performance.
• #21: The indexed polygons can be retrieved by the previously shown bounding box and distance searches. We have also recently added a geo-shape search type that allows to search for shapes that are related with other shapes (and points). The currently supported spatial relations are intersects, is within, and contains. It is also possible to apply transformations to the shapes used in the search. This allows to build complex shapes in search-time from other WKT shapes.
• #22: In this example we are searching for all the indexed shapes within a triangle. In this case we search for places within the Bermuda Triangle. Please note how we define both, the spatial operation to be applied, and the format of the search shape, which is WKT. In addition, we can recursively define the search shape as transformations of other shapes, as we will see in the Nephila's business use cases that Jonathan will show us.
• #23: Thank you Andres, I am Jonathan Nappée, I have been working with Nephila Capital for a year now in the Bermuda office. I first started talking to Andres while trying to solve some of our geospatial challenges with Stratio Lucene index in Cassandra. It already contained basic point indexing and distance search features but I had more complex indexing and search cases to implement. It turned out Andres also wanted to improve this aspect of the index. So when we met in London in February we very quickly came up with this idea of transformations. I want to show you now a couple of simplified examples how features could be used in the context of Nephila.
• #24: To begin with, let me introduce Nephila Capital and briefly explain it’s business. We are an investment fund that specializes in natural catastrophe property insurance. That means we deal with house or building insurances against hurricane, earthquakes or flood kind of disaster. As you can imagine we manipulate many different kinds of geospatial data sets, from properties we insure to the natural catastrophes impacts.
• #25: Let me explain now the setup for these examples. We started by indexing all US census blocks, that is to say shapes of blocks of buildings, inside a Cassandra cluster. Some of the block shapes are very complex and can contain hundreds of points and multiple polygons. To improve the efficiency we first tried indexing the convex hull but then switched to a composite indexing strategy. I’ll go into more details on this strategy in a few slides. We also indexed all US fire and police stations locations.
• #26: This is how the blocks table looks like. Each block contains its shape, centroid location, state income ratio and other information. We then indexed the different fields and in particular the block’s shapes.
• #27: This is how the stations table looks like. Each station contains its shape, centroid location, state, city and other information. We then indexed the different fields and in particular the locations.
• #28: Before I start showing the examples, let me explain a bit more the composite indexing strategy. This strategy uses two separate index structures, one to achieve speed and other to achieve acuracy. The first search structure is a geohash recursive prefix tree, usually with low precision. This geohash tree is used to quickly discard most of not matching documents. Then, the second search structure, which is a simple covering index, is used to discard the false positives produced by the geohash tree. This composite approach allows to quickly retrieve results with complete accuracy while keeping the index relatively small, and it is specially usefull with our dataset, which is composed by very complex polygons that would produce too much terms in a regular high-accuracy geohash search tree.
• #29: For our first use case we want to perform a search of blocks that intersect with a given shape. This is an important feature for us as the most damaged properties when a hurricane landfall happens are usually the ones closest to the shore. We can index the coast line but we usually want to see different buffer zones, the 1km, the 5 km and so on. So we use this buffer transformation to give us this flexibility.
• #30: In this second use case, we are interested in finding which blocks are far from any police or fire station. A property far from a fire station will probably on average suffer more damage from a fire than one close to a station. Thus we can use that information in evaluating the insurance risk. We consider that the beyond an 8 miles radius the fire station response becomes longer and thus less efficient.
• #31: So we define a rectangular zone of interest in which we find all the fire stations. We then build the 8 miles radius shapes from the fire stations and merge them. Finally we search for all blocks in the rectangular zone, not within the stations safer zone.
• #32: For the last use case we consider that a Hurricane just hit the US, in this case we use Hurricane Katrina’s example. We know every 6 hours the location of the hurricane and two zones, the larger zone with medium wind speeds and the smaller zone with highest wind speeds.
• #33: In this example we search for all the blocks in our portfolio of insurance that are hit by the hurricane. So we merge the two wind speed zones together, further merge with all the other shapes of the hurricane at different times and then look for the blocks inside.
• #36: And last but not least, our implementation is completely open source. It is published under the Apache License and it can be found at GitHub. We encourage you to take a look at it, and, of course, any contribution is more than welcomed.
• #37: Thanks for your attention! Any questions?