P9 speed of-light faceted search via oracle in-memory option by alexander tokarev

Editor's Notes

• #2: Hello, guys. My name is Alex. Have you ever used Oracle InMemory option to bring a performance profit? Perfect. Today we are going to discuss how we improved faceted search performance with Oracle InMemory option for one of our client.
• #3: Let me introduce myself I’m a performance architect and my main focus is data warehousing for big clients. I have huge Oracle experience but I works with Exasol and Tarantool databases as well.
• #4: My employer is a big global consulting company. We have a lot of famous clients in many industries and you know some of them probably .
• #5: We have rather straightforward agenda for today discussion We’ll discuss what is faceted search, Project overview and its performance issues I try to explain how Oracle InMemory option is implemented internally but without digging into details. My main focus is how we made the customer happy and share with you our issues you could face with as well For sure I’ll answer your questions.
• #6: Everybody knows what is faceted search. We have groups named facets, Values named constraints and Figures names facet counts
• #7: Common facet types are terms Intervals And ranges
• #8: What for do we need faceting We need facets if Users need to filter content using multiple taxonomy terms at the same time. Users want to combine text searches, taxonomy term filtering, and other search criteria. trying to discover relationships or trends between contents. Navigate inside tons of data And your end users are already frightened by "advanced" search forms in enterprise applications
• #9: There are 2 common approaches how to store data for facets – Tag-based where we store terms facets in a list of delimited somehow values Or Plain-based, where we store all values in a wide table. It is used for range facets mostly.
• #10: The fist option to implement faceted search is tags. Everyone knows what is tags but let me remind. Tag is a keyword assigned to an object Mostly is is chosen informaly If it is assigned by the viewer and tags count is unlimited it is folksonomy When assigned by the creator and a set of tags is limited it is taxonomy
• #11: We use taxonomy and foksonomy simoultaneously, so the best part of our facets are terms and we chosen tag-based storage.
• #12: Let me show some statistics. We extract objects, tags, facets for 2 years We have about 3 000 000 objects tagged, 42 000 000 terms applied, 100 000 unique terms, maximum tags count is 15 and there are about 150 facets. Is it whether decent volume or not? I think it is decent because we are bigger than StackOverflow by tags in 3 times.
• #13: We have ordinal architecture for enterprise. Web browser, balancer, a set of application servers which deal with inmemory grid and oracle with the replica on the backend. Pay attention that there is no full text search server which has been prohibited in accordance with customer policies.
• #14: We have microservice architecture and one of the services it tagging service which is responsible for data ingestion and faceted search. It is widely used by many other services and receives about 10000 requests per second. In order to serve search requests there are about 20 fine-tuned sql queries.
• #15: I can’t show you real printscreens from the application because of NDA so I implemented the mockup in Paint. As we could see UI is full of features as Predefined faceted search templates Full text search by random tags. We could search via audit information and we have faceted navigation for sure. We could filter out objects where some tags are present (not tag values) and user paging. It is rather sophisticated UI.
• #16: In terms of storage we store tags in Denormalized for in JSON and in a wide relational table for the sake of performance.
• #17: We could have a look into the table in details. It contains tagged object types, facet information, tags information and values. There is Denormalized object name and object description tags because they are used in full text search mostly.
• #18: Let’s have a look into performance metrics. I used logarithmic scale to put everything in one diagram. On a first glance figures look good. We do complicated searches less than zero point two seconds but actually it is bad because we need to serve more and more users and data so queries become slow
• #19: In accordance with one of my others presentations the best option for faceted search is full text search dedicated server but we had a lack of time to implement it as well as reluctant to introduce additional layers of complexity. Moreover, Oracle 18 has own faceted search implementation in Oracle Text feature1. We even did a proof of concept for the option and were happy but we use 12.1
• #20: We started looking into InMemory solutions especially regarding the customer already had Oracle InMemory license. In accordance with Oracle documentation it should be used for analytical queries which isn’t our case completely but we decided to try.
• #21: Why? 1. Because we were limited 2. The customer has a license 3. The table is rather wide, full of data which has a lot of equal values 4. So it will fit in memory especialy regarding compression features 5. And last but not least queries are full of aggregate functions
• #22: In accordance with the Oracle documentation InMemory feature is sort of silver bullets so we decided not waste time understanding how it works internally. What we knew that the option stores row and columnar data in dual mode maintaining their consistency automatically.
• #23: Let me remind you our table
• #24: We set InMemory area size to 10 gigabytes and assigned the table into the area. We decided that performance will be faster without compression so NO COMPRESSION option was used.
• #25: What is significant to stress that InMemory brought some overhead without compression
• #26: So we started our tests and faced with the difference in performance was extremely unsignificant which is expect for the search by primary keys
• #27: But for search by 1 tag
• #28: And search by 5 keys we don’t observe huge performance profit
• #29: When we started our POC we relied on Lufthansa usecase when they got average 21 performance profit but our profit was 10 times less. It means that we do something wrong and we need to understand how InMemory works internally.
• #30: Do you recall I mentioned InMemory area? It consists from 2 parts actually. The first one in IMCU – InMemory Compression Unit which stores data in columnar format And SMU – Snapshot Metadata Unit which stores transaction related metadata, dictionaries and Zone Maps. Do you know what is zone map?
• #31: Let’s have a look into simplified example. We have sales table which is split by 1 Mb IMCU blocks. Each block stores minimal and maximum values for each column so when we need to search sales for California we scans zone maps and check does our value is located in min/max boundaries so we do not scan extra blocks.
• #32: I converted SMU zone maps in readable format. As you could see they are stored per-column base as I told before. You see numeric zone maps and character ones. Please pay attention that dictionary_entries column contains only zero.
• #33: As we understood it happens because no compression were switched on.
• #34: Let’s enable the compression
• #35: We’ve seen that dictionary extries had been populated which should give dictionary-based pruning in accordance with the documentation I presume it happened because of zone maps have all values from A to Z in one IMCU. Moreover I haven’t found any statistics which could distinguish CU pruning reason: is it whether dictionary based or Zone maps based
• #36: We’ve seen that dictionary entries had been populated which should give dictionary-based pruning in accordance with the documentation Moreover the screenshot gives us very interesting case where zone maps could not be used. Try to guess why. Because the IMCU stores all possible character values so the pruning will not be efficient at all.
• #37: In order to fix I found 2 tricks: the old on and the new one. Any ideas what I’m talking about? The trick is arrange data in a proper fashion by the field we use for search
• #38: Compression ratio is rather high but it depends from you data for sure. You could see that compression doesn’t influence on ingestion so high compression could be used as the default. Frankly speaking loading 6 gigabytes for 1 minute it is to much so we decided to have a look into FastStart feature.
• #39: The feature writes InMemory area in a columnar format and once database is rebooted it loads the data in memory with the speed of file reading so we got 3 times loading profit.
• #40: It is time for rerun our tests. Try to guess which performance profit we got once we used high compression ration? Actually no profit at all. Let’s continue digging in details
• #41: Let’s have a look into one of simplified SQL used for faceted search. It has some aggregate functions and group by clause.
• #42: In order to improve facet count and search calculation InMemory aggregations should be used. They create Key vector which is used in join process as well as doing aggregate functions calculation on the fly.
• #43: If we recall denormilized table structure we identify that the best part of fields has binary type because we use global unique identifiers as keys so Zone maps aren’t used Key vector transformation replaced by Bloom filter And the same happens if keys are varchar
• #44: In order to address the challenge we decided to use Join groups. It is sort of Join cluster which I’ve never seen in production though. Join groups are intended to implement join without decompressioning InMemory Compression Unit based on dictionary entries.
• #45: Let’s create join group. There is no statistics to understand whether join groups were used or not so it is required to parse Real Time Sql monitor output to achieve it
• #46: After some investigations we decided not to use join groups because of They work with simplest join conditions only Compression ratio should be equal Only simple data types are required, joins by big varchar2 fields don’t work If a lot of information is loaded into InMemory tables join groups are ignored More PGA is required by sessions FastStart refuse to work
• #47: So we changed all foreign keys field data types to number which required changes in our application and we removed all indexes
• #48: We understood that no reasons to store all columns inmemory because we search by them via Oracle Full Text Search indexes
• #49: So the final structure is simple data types without b-tree indexes and 2 fields are removed from inmemory storage
• #50: We did some performance tests and found sporadic search performance degradation which happens where records were inserted or updated.
• #51: Let’s understand how Oracle provides read consistency for InMemory. IM introduces third mechanyzm of read consistency: the first one is the ordinal one with rollback segments, the second is latches for Result Cache and InMemory has the mechanism very close to Postgress Once a record is changed it is marked as stale. All stale records are read either from buffer cache or disk. Oracle repopulates stale records by staleness threshold or each 2 minutes by so named trickle repopulation process. It could be done in parallel plus we could state how much resources could be consumed for stale records repopulation
• #52: I suspect these parameters are responsible for IMCU repopulation but substring search gives us about 34 undocumented parameters so I’m reluctant to change any of them
• #53: In order to understand where data were scanned from there are 2 statistics: Scan row cache And Scan block cache Both of them should be equal to zero
• #54: In order to look after repopulation process use v$im_smu_head view.
• #55: The second view is v$im_header. Let’s have a look into details – there is very interesting case here. We repopulate small number of blocks with trickle repopulation process initially but in 2 seconds We run other trickle repopulation process which identified that too much of rows is affected and it is better to repopulate data block completely.
• #56: Let’s have a look what we have by default Oracle allocates 2 parallel servers and spends 1 percent of resource for trickle process
• #57: As soon as we set 4 repopulation servers and 8 percent of time to repopulate stale records and performance spikes disappeared
• #58: InMemory option is very easily to trace and maintain but let me share a secret information with you. We needn’t to know all of them at all.
• #59: 8 parameters covers the best part of InMemory performance tuning. Let me share one more secret. only 1 parameter is enough. Try to guess which one and why I could state the query I took these statistics from is non-efficient It states how efficient zone maps indexes are used.
• #60: So 3 settings covers the best part of dba activities
• #61: And 2 views give us enough information for basic InMemory undertanding
• #62: Let’s have a look into final figures. No performance profit for PK scan which is expected.
• #63: Let’s have a look into final figures. With proper InMemory implementation we have 4 times performance profit for one tag search
• #64: And 5 times performance profit for search by 5 tags so it is time moving to conclusion
• #65: Let’s elaborate DBA findings. InMemory area size should be estimated taking into account compression and your data If we have all data in memory it doesn’t mean we are extremely fast InMemory takes time to be loaded after server reboot 8 statistics is enough to help developers with the best part of their issues Allign trickle processes settings with your workload
• #66: What we’ve fount that in our usecase extremely advanced inmemory features like join groups, inmemory aggregations hasn’t brought significant performance profit. Zone maps work perfect with numeric and date data types. Dictionary based pruning doesn’t work in our oracle version at least. More simple data types are used – more performance we have It is extremely significant that even high compression doesn’t affect ingestion speed Different read consistency model leads that even rollback statements could slow down your queries
• #67: So our plan is rather straightforward: Add memory Test the main purpose the InMemory was intended for Try to understand all statistics Enable FastStart on production Play with Oracle 18 cool features like lightweight parallel InMemory scans
• #68: Our main conclusion is that you shouldn’t rely on documentation and experts – try and measure your particular use-case Which could prove that IM works with short queries as well You must understand how it works internally otherwise you will not get performance profit Unfortunately application was changed But extra layers, message queues, hardware need to be added We had managed to put our IM POC in production very fast (1 week or so) And business got 4 times performance boost. Spending a lot of money That’s it from my side for today so
• #69: Thank you for your time.