SlideShare a Scribd company logo

Drilling Deep Into Exadata Performance

E
Enkitec

The document discusses techniques for analyzing SQL performance on Oracle Exadata systems using tools like ASH, SQL monitoring, and ExaSnapper. It provides examples of using these tools to identify SQL statements that are not optimized for Exadata's Smart Scan feature and determining if problematic statements are long-running queries or frequent short queries. The document also demonstrates how to selectively force full table scans for reporting workloads while keeping indexes available for OLTP workloads.

Technology
1 of 35
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Drilling  Deep  into  Exadata  Performance  with  ASH,   SQL  Monitoring  and  ExaSnapper   Tanel  Põder   Enkitec  Europe     h.p://www.enkitec.com   h.p://blog.tanelpoder.com   www.enkitec.com     1      
Intro:  About  me   •  Tanel  Põder   •  Former  Oracle  Database  Performance  geek   •  Present  Exadata  Performance  geek  ;-­‐)   •  My  Exadata  experience   •  I  have  had  the  luck  to  work  with  all   possible  Exadata  configuraMons  out  there   •  Exadata  V1  …  X2-­‐8   •  MulM-­‐rack  Exadatas  …   •  Even  a  mixed  rack  Exadata  (V2  <-­‐>  X2-­‐2  :)   Expert  Oracle  Exadata   book   (with  Kerry  Osborne  and   Randy  Johnson  of  Enkitec)   www.enkitec.com     2      
Agenda   1.  A  li.le  detour  1:  Finding  non-­‐Exadata  friendly  SQL   2.  A  li.le  detour  2:  selecMvely  forcing  full  table  scans  in  mixed   workload  environments!   3.  A  li.le  bit  of  Smart  Scan  internals   4.  Reading  a  SQL  Monitoring  report  on  Exadata   •  …  and  where  it  falls  short   5.  Using  advanced  Exadata  performance  metrics   •  Exadata  Snapper  (ExaSnap)!!!   www.enkitec.com     3      
Detour  1     Finding  non-­‐Exadata-­‐friendly  SQL   www.enkitec.com     4      
1)  Finding  top  non-­‐Exadata-­‐friendly  SQLs   •  OpMon  1:   •  Find  SQLs  which  wait  for  non-­‐smart  scan  IO  operaMons  the  most   •  ASH!   •  OpMon  2:   •  Find  SQLs  doing  the  most  disk  reads  without  Smart  Scans   •  Highest  MB  read  or  highest  IOPS   a)  ASH!   –  SUM(DELTA_READ_IO_REQUESTS)   –  SUM(DELTA_READ_IO_BYTES)   b)  ..or  join  to  V$SQLSTAT  (or  DBA_HIST_SQLSTAT)   –  SUM(PHYSICAL_READ_REQUESTS_DELTA)   –  SUM(PHYSICAL_READ_BYTES_DELTA)   www.enkitec.com     5      
2)  Are  these  long-­‐running  or  frequently  executed  short   queries?   •  Exadata  Smart  scans  are  not  opMmized  for  ultra-­‐frequent  execuMon   •  Smart  Scanning  1000s  of  Mmes  per  second  isn’t  good   •  That’s  why  Smart  Scans  are  not  good  for  your  OLTP-­‐queries   •  Here’s  an  analogy:   1.  Want  to  deliver  a  single  parcel  to  a  desMnaMon   •  Use  a  Ferrari   2.  Want  to  deliver  10  000  parcels  to  a  desMnaMon   •  Use  a  truck   3.   Want  to  deliver  10  000  000  parcels  to  a  desMnaMon   •  Use  a  freight  train   •  Smart  Scan  is  the  Exadata’s  freight  train   •  Brute  force  scanning  with  relaMvely  high  inerMa  to  get  started,  not  a  few   quick  (buffered)  I/O  operaMons  done  with  surgical  precision   www.enkitec.com     6      
Demo  –  exafriendly.sql   •  Drill  down  into  ASH  wait  data:   SQL> @exadata/exafriendly.sql gv$active_session_history SESSION WAIT_CLASS EVENT SECONDS PCT ------- -------------------- ---------------------------------------- ---------- ------ ON CPU 192356 34.6 WAITING User I/O cell single block physical read 191838 34.5 WAITING User I/O db file parallel read 40577 7.3 WAITING User I/O cell smart table scan 28593 5.1 WAITING User I/O cell multiblock physical read 19424 3.5 WAITING User I/O direct path read temp 18398 3.3 WAITING Application enq: RO - fast object reuse 8690 1.6 WAITING User I/O direct path read 5812 1.0 ... PLAN_LINE USERNAME EVENT SECONDS PCT ------------------------------------- ------------ ---------------------------------- --------- ------ TABLE ACCESS BY LOCAL INDEX ROWID USER_104 cell single block physical read 40954 21.3 TABLE ACCESS BY INDEX ROWID USER_779 cell single block physical read 32129 16.7 INDEX RANGE SCAN USER_104 cell single block physical read 25272 13.2 TABLE ACCESS STORAGE FULL USER_49 cell single block physical read 9258 4.8 INDEX RANGE SCAN USER_779 cell single block physical read 4118 2.1 TABLE ACCESS STORAGE FULL USER_783 cell single block physical read 3641 1.9 UPDATE USER_104 cell single block physical read 3509 1.8 TABLE ACCESS BY INDEX ROWID USER_420 cell single block physical read 3341 1.7 MERGE USER_962 cell single block physical read 2910 1.5 www.enkitec.com     7      
Demo  –  mon_topsql.sql   •  TOP  Time-­‐consuming  SQLs  with  IO  &  execuMon  counts   •  That  way  we’ll  separate  the  Ferraris  from  Freight  Trains!   •  The  “Ferraris”  aren’t  Exadata  smart  scan  friendly   www.enkitec.com     8      
Other  sources  for  overview  informaMon   •  ASH  reports  (based  on  ASH  data  which  I’ve  used)   •  EM  12c  ASH  analyMcs   www.enkitec.com     9      
Detour  2         How  to  force  full  table  scans  for  reports?   …while  keeping  the  OLTP  workloads  free  to  use  indexes?     1)  Make  all  indexes  invisible   2)  Set  op<mizer_use_invisible_indexes  =  true  for  OLTP  sessions     •  Stay  tuned  for  my  whitepaper  “SelecMve  Index  Visibility  for  Oracle  Exadata”     on  this  topic  :-­‐)   www.enkitec.com     10      
Exadata  Architecture   •  All  DB  nodes  talk  to  all  (configured)  cells  -­‐  ASM  striped  data   •  A  cell  never  talks  to  another  cell  !!!   DB  Node  1   DB  Node  2   DB  Node  3   DB  Node  4   Cell  1   Cell  2   Cell  3   Cell  4   Cell  5   Cell  6   Cell  7   www.enkitec.com     11      
Smart  Scans:  Asynchronous,  independent  prefetching   PARSING IN CURSOR #47233445473208 len=38 dep=0 uid=93 oct=3 lid=93 tim=1303079537221426 select * from t3 where owner like 'S%' END OF STMT PARSE #47233445473208:c=1000,e=8964,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=4161002650,ti EXEC #47233445473208:c=0,e=21,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4161002650,tim=1303 WAIT #47233445473208: nam='SQL*Net message to client' ela= 4 driver id=1413697536 #bytes WAIT #47233445473208: nam='SQL*Net more data to client' ela= 16 driver id=1413697536 #by WAIT #47233445473208: nam='reliable message' ela= 1337 channel context=11888341784 chann WAIT #47233445473208: nam='enq: KO - fast object checkpoint' ela= 143 name|mode=12634685 WAIT #47233445473208: nam='enq: KO - fast object checkpoint' ela= 130 name|mode=12634685 WAIT #47176529789912: nam='cell smart table scan' ela= 25 cellhash#=3176594409 p2=0 p3=0 WAIT #47176529789912: nam='cell smart table scan' ela= 882 cellhash#=2133459483 p2=0 p3= WAIT #47176529789912: nam='cell smart table scan' ela= 34 cellhash#=3176594409 p2=0 p3=0 WAIT #47176529789912: nam='cell smart table scan' ela= 837 cellhash#=2133459483 p2=0 p3= WAIT #47176529789912: nam='cell smart table scan' ela= 26 cellhash#=2133459483 p2=0 p3=0 WAIT #47176529789912: nam='cell smart table scan' ela= 22 cellhash#=379339958 p2=0 p3=0 The  waits  are  so  short  due  to  the   The  object  checkpoint-­‐ If  cell  smart  table/ asynchronous  nature  of  smart   related  wait  events  reliable   index  scan  waits   scans.  Cellsrvs  in  each  cell  process   message  and  enq:  KO  –  fast   show  up,  then  a   the  blocks  independently  to  fill   object  checkpoint  always   smart  scan  is   their  send  buffers  and  the  DB  just   precede  every  direct  path   a.empted   pulls  the  results  from  there   scan  (thus  smart  scan  too)   www.enkitec.com     12      
Storage  cells  are  “shared  nothing”   •  And  they  don’t  see  what’s  happening  in  the  database  layer…   Database  Layer   A  large  table   Smart  Scan   Cell  Layer   Cell  1   Cell  2   Cell  3   Cell  4   Cell  5   Cell  6   Cell  7   www.enkitec.com     13      
Physical  disks  -­‐  Simple  math   •  SequenMal  “brute  force”  scan  rate  150  MB/sec  per  disk   or     •  200  random  IOPS  per  disk   I’m  leaving   flash  cache   •  12  disks  in  a  storage  cell   out  for   simplicity  for   •  14  storage  cells  in  a  full  rack   now   •  150  *  12  *  14  =  25200  MB/sec  physical  disk  scanning  rate   •  If  doing  only  sequenMal  brute  force  scanning     However,   Index  scans   •  200  *  12  *  14  *  8kB  =  262.5  MB/sec   can  read   only  a  subset   •  Random  physical  read  rate  (index  scan!)  around  100x  slower!   of  data   www.enkitec.com     14      
The  MoMvaMon  for  WriMng  Exadata  Snapper   [celladmin@enkcel01 ~]$ cellcli CellCLI: Release 11.2.2.4.0 - Production on Mon Mar 05 08:19:41 CST 2012 Copyright (c) 2007, 2011, Oracle. All rights reserved. Cell Efficiency Ratio: 699 CellCLI> ????   Following  just  the  Cell  Efficiency  RaVo  can  be  as   misleading  as  tuning  by  Buffer  Cache  Hit  RaVo!   ?   www.enkitec.com     15      
Processing   Data  ReducMon,  IO  Avoidance,  Early  Filtering   Stage   SorMng,   aggregaMon,   joins  etc   Smart  Scan  returned  MB   +  Extra  block  reads   Temp  IO   Bugs,  chained   Smart  Scan  returned  MB   +  Extra  block  reads   rows,  migrated   rows,  read   Smart  Scan  returned  MB   consistency   Spinning  Disk  Read  MB   Flash  Reads   Storage   indexes   “Disk”  Read  MB   Reads  avoided                          Compressed  Data   Uncompressed  Data   Data  Volume   www.enkitec.com     16      
NegaMve  Cell  Offload  Efficiency  raMo?   •  Must  understand  where  does  this  raMo  come  from   •  RaMo  of  which  Exact  metrics?   •  And  use  those  metrics  in  the  future   www.enkitec.com     17      
NegaMve  Cell  Offload  Efficiency:  Data  Load  Example   Why  6  GB?   2  +  2  =  4  GB   The  “Bytes  returned  by  Exadata”   So,  write  IOs  also  influence  cell  offload   metric  actually  uses  the  “cell  physical   efficiency  calculaMon  (data  loads,  sort/join/ IO  interconnect  bytes”  metric   aggregate  operaMon  TEMP  usage).     internally,  which  includes  all  traffic,  not   Write  IOs  are  double/triple  mirrored  by   just  rows  returned  from  smart  scan.     ASM!   www.enkitec.com     18      
InterpreMng  the  raw  metrics  with  Exadata  Snapper   •  IO  Efficiency  breakdown   •  How  much  physical  IO  did  we  really  do?   •  How  much  reads  /  how  much  writes?   •  How  much  disk  IO  did  we  avoid  doing  thanks  to  Flash  Cache?   •  How  much  disk  IO  did  we  avoid  doing  thanks  to  Storage  Indexes?   •  What  was  the  total  interconnect  traffic?   •  How  much  data  was  fully  processed  in  the  cell  (and  not  shipped  back   in  blocks  due  to  fallback  or  problems  in  the  cell?)   •  How  much  data  was  returned  by  a  Smart  Scan  (as  rows)?   •  @exadata/exasnap.sql   •  Beta  quality   •  Meant  to  complement  ASH  and  SQL  Monitoring  reports,  not  replace   them   www.enkitec.com     19      
ExaSnap  example:  A  CTAS  statement   SQL> @exadata/exasnap % 123 124 ----------------------------------------------------------------------------------------------------------------- -- Exadata Snapper v0.5 BETA by Tanel Poder @ Enkitec - The Exadata Experts ( http://www.enkitec.com ) ----------------------------------------------------------------------------------------------------------------- SID CATEGORY METRIC IOEFF_PERCENTAGE MB ------ -------------------- ----------------------- ---------------------------------------------------- ---------- 1280 DB_IO DB_PHYSIO_MB |################################## | 4435 DB_IO DB_PHYSRD_MB |################# | 2219 DB_IO DB_PHYSWR_MB |################# The  real  disk  writes   | 2216 are  doubled  due  to   AVOID_DISK_IO PHYRD_FLASH_RD_MB | | 2 AVOID_DISK_IO PHYRD_STORIDX_SAVED_MB | ASM  double-­‐mirroring   | 0 DISK_IO SPIN_DISK_IO_MB |##################################################| 6649 DISK_IO SPIN_DISK_RD_MB |################# | 2217 DISK_IO SPIN_DISK_WR_MB |################################## | 4432 REDUCE_INTERCONNECT PRED_OFFLOAD_MB |################# | 2216 REDUCE_INTERCONNECT TOTAL_IC_MB |##################################################| 6444 REDUCE_INTERCONNECT SMART_SCAN_RET_MB |################ | 2009 REDUCE_INTERCONNECT NON_SMART_SCAN_MB |################################## | 4435 CELL_PROC_DEPTH CELL_PROC_DATA_MB |################# | 2232 CELL_PROC_DEPTH CELL_PROC_INDEX_MB | | 0 www.enkitec.com     20      
ExaSnap  example  2:  Storage  Index  Savings   The  real  (spinning)  disk  reads  IO  was  only   1078  MB  thanks  to  1138  MB  of  disk  IO   SQL> @exadata/exasnap basic 90 91 avoided  due  to  storage  indexes:   ----------------------------------------------------------------------------------------------------------------- (2216  –  1138  =  1078)   -- Exadata Snapper v0.5 BETA by Tanel Poder @ Enkitec - The Exadata Experts ( http://www.enkitec.com ) ----------------------------------------------------------------------------------------------------------------- CATEGORY METRIC IOEFF_PERCENTAGE MB -------------------- ----------------------- ---------------------------------------------------- ---------- ---- DB_LAYER_IO DB_PHYSIO_MB |##################################################| 2216 DB_LAYER_IO DB_PHYSRD_MB |##################################################| 2216 DB_LAYER_IO DB_PHYSWR_MB | | 0 AVOID_DISK_IO PHYRD_FLASH_RD_MB | | 0 AVOID_DISK_IO PHYRD_STORIDX_SAVED_MB |########################## | 1138 REAL_DISK_IO SPIN_DISK_IO_MB |######################## | 1079 REAL_DISK_IO SPIN_DISK_RD_MB |######################## | 1078 REAL_DISK_IO SPIN_DISK_WR_MB | | 1 REDUCE_INTERCONNECT PRED_OFFLOADABLE_MB |##################################################| 2216 REDUCE_INTERCONNECT TOTAL_IC_MB | | 2 REDUCE_INTERCONNECT SMART_SCAN_RET_MB | | 2 REDUCE_INTERCONNECT NON_SMART_SCAN_MB | | 0 CELL_PROC_DEPTH CELL_PROC_DATA_MB |######################## | 1078 CELL_PROC_DEPTH CELL_PROC_INDEX_MB | | 0 All  1078  MB  worth  of  blocks  got  offloaded:   they  were  opened  and  processed  inside  the   cells  (data  layer)   www.enkitec.com     21      
So,  why  isn’t  my  query  Exadata-­‐friendly?   www.enkitec.com     22      
Demo  –  Drilling  down  into  a  SQL  execuMon   1.  SQL  Monitoring  report   •  ExecuMon  plan!   •  Where  is  most  of  the  response  Mme  spent  (retrieval  vs.  subsequent   processing)   •  Are  smart  scans  used  for  data  retrieval?   •  IO  MB  read  from  disks  vs  data  returned  from  the  cells   •  (also  called  the  offload  efficiency  raMo  but  knowing  the  underlying   numbers  is  way  be.er)     2.  ExaSnapper   •  Or  read  the  raw  v$sesstat  metrics  if  you  dare  ;-­‐)   www.enkitec.com     23      
Warm-­‐up  case  study   •  TesMng  a•er  migraMon,  a  query  is  slow:   cell  single  block  physical  read   wait  events  in  a  parallel  query?   www.enkitec.com     24      
Warm-­‐up  case  study:  check  the  execuMon  plan   •  Parallel  execuMon  plan  does  not  force  full  table  scans...   ------------------------------------------------------------------------------------------------------ | Id | Operation | Name | TQ |IN-OUT| PQ Distrib | ------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | | | | | 1 | PX COORDINATOR | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10002 | Q1,02 | P->S | QC (RAND) | |* 3 | FILTER | | Q1,02 | PCWC | | | 4 | HASH GROUP BY | | Q1,02 | PCWP | | | 5 | PX RECEIVE | | Q1,02 | PCWP | | | 6 | PX SEND HASH | :TQ10001 | Q1,01 | P->P | HASH | | 7 | NESTED LOOPS | | Q1,01 | PCWP | | | 8 | NESTED LOOPS | | Q1,01 | PCWP | Serial  stage   | | 9 | NESTED LOOPS | | Q1,01 | PCWP | in  a  parallel   | | 10 | BUFFER SORT | | Q1,01 | PCWC | plan   | | 11 | PX RECEIVE | | Q1,01 | PCWP | | | 12 | PX SEND ROUND-ROBIN | :TQ10000 | | S->P | RND-ROBIN | |* 13 | TABLE ACCESS BY GLOBAL INDEX ROWID| ORDERS | | | | |* 14 | INDEX RANGE SCAN | ORD_STATUS_IX | | | | | 15 | PARTITION HASH ITERATOR | | Q1,01 | PCWP | | Parallel  index   |* 16 | TABLE ACCESS BY LOCAL INDEX ROWID | CUSTOMERS | Q1,01 | PCWP | scans  (|on   |* 17 | INDEX UNIQUE SCAN | CUSTOMERS_PK | Q1,01 | PCWP | | | 18 | PARTITION HASH ITERATOR | | Q1,01 | PCWP | different   | |* 19 | INDEX RANGE SCAN | ORDER_ITEMS_PK | Q1,01 | PCWP | parMMons)   | | 20 | TABLE ACCESS BY LOCAL INDEX ROWID | ORDER_ITEMS | Q1,01 | PCWP | | ------------------------------------------------------------------------------------------------------ www.enkitec.com     25      
Warm-­‐up  case  study  –  adjusted  execuMon  plan   •  A•er  forcing  full  table  scans:   ------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Pstart| Pstop | Fully  parallel   TQ |IN-OUT| PQ Distrib | execuMon   ------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | | plan   | | 1 | PX COORDINATOR | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10003 | | | Q1,03 | P->S | QC (RAND) | |* 3 | FILTER | | | | Q1,03 | PCWC | | | 4 | HASH GROUP BY | | | | Q1,03 | PCWP | | | 5 | PX RECEIVE | | | | Q1,03 | PCWP | | | 6 | PX SEND HASH | :TQ10002 | | | Q1,02 | P->P | HASH | |* 7 | HASH JOIN | | | | Q1,02 | PCWP | | | 8 | PART JOIN FILTER CREATE | :BF0000 | | | Q1,02 | PCWP | | | 9 | PX RECEIVE | | | | Q1,02 | PCWP | | | 10 | PX SEND BROADCAST | :TQ10001 | | | Q1,01 | P->P | BROADCAST | |* 11 | HASH JOIN | | | | Q1,01 | PCWP | | | 12 | PART JOIN FILTER CREATE | :BF0001 | | | Q1,01 | PCWP | | | 13 | PX RECEIVE | | | | Q1,01 | PCWP | | | 14 | PX SEND BROADCAST | :TQ10000 | | | Q1,00 | P->P | BROADCAST | | 15 | PX BLOCK ITERATOR | | 1 | 16 | Q1,00 | PCWC | | |* 16 | TABLE ACCESS STORAGE FULL| ORDERS | 1 | 16 | Q1,00 | PCWP | | | 17 | PX BLOCK ITERATOR | |:BF0001|:BF0001| Q1,01 | PCWC | | |* 18 | TABLE ACCESS STORAGE FULL | CUSTOMERS |:BF0001|:BF0001| Q1,01 | PCWP | Full  table   | | 19 | PX BLOCK ITERATOR | |:BF0000|:BF0000| Q1,02 | PCWC | (parMMon)   | |* 20 | TABLE ACCESS STORAGE FULL | ORDER_ITEMS |:BF0000|:BF0000| Q1,02 | PCWP | | scans  with   ------------------------------------------------------------------------------------------------------------- 18 - storage(:Z>=:Z AND :Z<=:Z AND ("C"."NLS_TERRITORY"='AMERICA' AND 
 bloom  filter   SYS_OP_BLOOM_FILTER(:BF0000,"C"."CUSTOMER_ID"))) offloading   www.enkitec.com     26      
Case  2:  Response  Mme  24  seconds  –  where  is  it  spent?   1)  Full  scan:  TABLE  ACCESS  STORAGE  FULL   Smart  scan   will  help!   2)  But  waiMng  for  a  buffered  read  wait  event   •  cell  mul<block  physical  read  instead  of  cell  smart  table/index  scan   www.enkitec.com     27      
Case  2:  The  same  query  runs  in  3  seconds  with  Smart  Scan   •  So,  why  is  it  faster?   •  Data  retrieval  (ACCESS)  from  storage  is  faster   www.enkitec.com     28      
Case  2:  Now  let’s  do  something  with  the  data…   •  Same  query,  with  an  extra  GROUP  BY:   •  Takes  3.8  minutes!  (228  seconds)   •  Why?  Let’s  see  what’s  taking  the  extra  Mme:   www.enkitec.com     29      
Checkpoint   •  Smart  Scans  make  the  data  retrieval  from  storage  faster   •  Any  other  problems  require  the  usual  SQL  &  DB  opMmizaMon   www.enkitec.com     30      
Smart  Scans  do  not  make  any  of  these  things  faster  (by  design):   •  Index  unique/range/skip  scans  *     *  A  prerequisite  for   smart  scans  is  a  full   •  SorMng   segment  scan  anyway   •  AggregaMon  (GROUP  BY)   •  AnalyMc  FuncMons   •  Filters  that  can’t  be  pushed  down  to  table  level:   •  WHERE  t1.col  +  t2.col  =  1   •  Any  funcMon  calls  in  projecMon  (select  list)   •  PL/SQL  funcMon  calls  in  projecMon  (or  WHERE  clause)   •  Nested  loop  joins,  sort-­‐merge  joins  and  FILTER  lookups   •  Hash  joins  are  special  though   •  So,  you’ll  have  to  see  where  is  your  bo.leneck!   This  is  not  a  full   •  SQL  Monitoring  report  is  a  good  start   list  of  limitaMons.   www.enkitec.com     31      
A  query  bo.lenecked  by  data  processing,  not  retrieval   •  A  SQL  performing  data  load  and  spills  to  TEMP   www.enkitec.com     32      
Case  3:  Case  insensiMve  search   ALTER SESSION SET nls_comp = LINGUISTIC; The  plan  hash  value  does  not  change  as  it  doesn’t   ALTER SESSION SET nls_sort = BINARY_CI; take  the  predicate  differences  into  account!   SELECT SUM(credit_limit) FROM soe.customers SELECT SUM(credit_limit) FROM soe.customers WHERE cust_first_name LIKE 'j%' WHERE cust_first_name LIKE 'j%' Plan hash value: 296924608 Plan hash value: 296924608 FAST ~ 2 seconds SLOW ~14 seconds ----------------------------------------------- ----------------------------------------------- | Id | Operation | Name | | Id | Operation | Name | ----------------------------------------------- ----------------------------------------------- | 0 | SELECT STATEMENT | | | 0 | SELECT STATEMENT | | | 1 | SORT AGGREGATE | | | 1 | SORT AGGREGATE | | |* 2 | TABLE ACCESS STORAGE FULL| CUSTOMERS | |* 2 | TABLE ACCESS STORAGE FULL| CUSTOMERS| ----------------------------------------------- ----------------------------------------------- Predicate Information Predicate Information --------------------------------------------- ---------------------------------------------- 2 - storage("CUST_FIRST_NAME" LIKE 'j%') 2 - filter(NLSSORT filter("CUST_FIRST_NAME" LIKE 'j%') ("OWNER",'nls_sort=''BINARY_CI''') >HEXTORAW('7300’) ) Where’s  the   storage  predicate?   www.enkitec.com     33      
FAST  –  2  seconds   Case3:   SLOW  –  14  seconds   www.enkitec.com     34      
Thank  you  –  and  oh,  wait!   •  Advanced  Exadata  Performance  seminar!   •  By  Tanel  Poder   •  SystemaMc  Exadata  Performance  TroubleshooMng  and  OpMmizaMon   •  3-­‐day  seminar  available  from  November   •  We’ll  go  very  deep!     •  QuesMons:  tanel@tanelpoder.com   •  Blog:    h.p://blog.tanelpoder.com   •  ConsulMng:  h.p://www.enkitec.com   www.enkitec.com     35      

More Related Content

PDF
Tuning SQL for Oracle Exadata: The Good, The Bad, and The Ugly Tuning SQL fo...
Enkitec
 
PDF
Parallel Query on Exadata
Enkitec
 
PPTX
Understanding Query Optimization with ‘regular’ and ‘Exadata’ Oracle
Guatemala User Group
 
PPTX
Indexing in Exadata
Enkitec
 
PPTX
Oracle Exadata X2-8: A Critical Review
Texas Memory Systems, and IBM Company
 
PDF
Managing Exadata in the Real World
Enkitec
 
PDF
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Nelson Calero
 
PPTX
Sun Oracle Exadata Technical Overview V1
jenkin
 
Tuning SQL for Oracle Exadata: The Good, The Bad, and The Ugly Tuning SQL fo...
Enkitec
 
Parallel Query on Exadata
Enkitec
 
Understanding Query Optimization with ‘regular’ and ‘Exadata’ Oracle
Guatemala User Group
 
Indexing in Exadata
Enkitec
 
Oracle Exadata X2-8: A Critical Review
Texas Memory Systems, and IBM Company
 
Managing Exadata in the Real World
Enkitec
 
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Nelson Calero
 
Sun Oracle Exadata Technical Overview V1
jenkin
 

What's hot (20)

PPTX
Christo kutrovsky oracle, memory & linux
Kyle Hailey
 
PDF
Exadata Patching Demystified
Enkitec
 
PPTX
Oracle Database 12c - The Best Oracle Database 12c Tuning Features for Develo...
Alex Zaballa
 
PDF
Tanel Poder - Scripts and Tools short
Tanel Poder
 
PDF
GLOC 2014 NEOOUG - Oracle Database 12c New Features
Biju Thomas
 
PDF
Mini Session - Using GDB for Profiling
Enkitec
 
PDF
Think Exa!
Enkitec
 
PPT
Sun Oracle Exadata V2 For OLTP And DWH
Mark Rabne
 
ODP
Exadata
talek
 
PPTX
Flex Cluster e Flex ASM - GUOB Tech Day - OTN TOUR LA Brazil 2014
Alex Zaballa
 
PPT
Ash masters : advanced ash analytics on Oracle
Kyle Hailey
 
PPTX
Scaling sql server 2014 parallel insert
Chris Adkin
 
PDF
In Memory Database In Action by Tanel Poder and Kerry Osborne
Enkitec
 
PDF
In Search of Plan Stability - Part 1
Enkitec
 
PDF
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Severalnines
 
PPTX
Ukoug15 SIMD outside and inside Oracle 12c (12.1.0.2)
Laurent Leturgez
 
PDF
Hotsos 2011: Mining the AWR repository for Capacity Planning, Visualization, ...
Kristofferson A
 
PPT
Intro to ASH
Kyle Hailey
 
PPT
Oracle Open World Thursday 230 ashmasters
Kyle Hailey
 
PDF
Exadata database machine_x5-2
MarketingArrowECS_CZ
 
Christo kutrovsky oracle, memory & linux
Kyle Hailey
 
Exadata Patching Demystified
Enkitec
 
Oracle Database 12c - The Best Oracle Database 12c Tuning Features for Develo...
Alex Zaballa
 
Tanel Poder - Scripts and Tools short
Tanel Poder
 
GLOC 2014 NEOOUG - Oracle Database 12c New Features
Biju Thomas
 
Mini Session - Using GDB for Profiling
Enkitec
 
Think Exa!
Enkitec
 
Sun Oracle Exadata V2 For OLTP And DWH
Mark Rabne
 
Exadata
talek
 
Flex Cluster e Flex ASM - GUOB Tech Day - OTN TOUR LA Brazil 2014
Alex Zaballa
 
Ash masters : advanced ash analytics on Oracle
Kyle Hailey
 
Scaling sql server 2014 parallel insert
Chris Adkin
 
In Memory Database In Action by Tanel Poder and Kerry Osborne
Enkitec
 
In Search of Plan Stability - Part 1
Enkitec
 
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Severalnines
 
Ukoug15 SIMD outside and inside Oracle 12c (12.1.0.2)
Laurent Leturgez
 
Hotsos 2011: Mining the AWR repository for Capacity Planning, Visualization, ...
Kristofferson A
 
Intro to ASH
Kyle Hailey
 
Oracle Open World Thursday 230 ashmasters
Kyle Hailey
 
Exadata database machine_x5-2
MarketingArrowECS_CZ
 
Ad

Viewers also liked (6)

PDF
Oracle Systems _ Jeff Schwartz _ Engineering Solutions Exadata - Exalogic.pdf
InSync2011
 
PPT
Collier exadata technical overview presentation 4 14-10
xKinAnx
 
PDF
Oracle Exadata Version 2
Jarod Wang
 
PDF
Epic Clarity Running on Exadata
Enkitec
 
PPT
Oracle infiniband cabling for Oracle appliances
Johan Louwers
 
PPT
Exadata x3 workshop
Fran Navarro
 
Oracle Systems _ Jeff Schwartz _ Engineering Solutions Exadata - Exalogic.pdf
InSync2011
 
Collier exadata technical overview presentation 4 14-10
xKinAnx
 
Oracle Exadata Version 2
Jarod Wang
 
Epic Clarity Running on Exadata
Enkitec
 
Oracle infiniband cabling for Oracle appliances
Johan Louwers
 
Exadata x3 workshop
Fran Navarro
 
Ad

Similar to Drilling Deep Into Exadata Performance (20)

PDF
Oracle Database In-Memory Option in Action
Tanel Poder
 
PPTX
Dev nexus 2017
Roy Russo
 
PPTX
Devnexus 2018
Roy Russo
 
PDF
Apache Spark At Scale in the Cloud
Rose Toomey
 
PDF
Apache Spark At Scale in the Cloud
Databricks
 
PDF
Spark on YARN
Adarsh Pannu
 
PDF
backgroundcommunicationandwaitevents-180124221026.pdf
ssuser785ce21
 
PDF
NoSQL with MySQL
FromDual GmbH
 
PDF
Brief Introduction to Parallella
Somnath Mazumdar
 
PDF
504_senegacnik_oracle_database_mutexes.pdf
cookie1969
 
PDF
My First 100 days with an Exadata (WP)
Gustavo Rene Antunez
 
PPTX
Real World Performance - Data Warehouses
Connor McDonald
 
PPTX
Performance Benchmarking: Tips, Tricks, and Lessons Learned
Tim Callaghan
 
PDF
Tos tutorial
manikainth
 
PPTX
The power of linux advanced tracer [POUG18]
Mahmoud Hatem
 
PDF
OOW09 Ebs Tuning Final
jucaab
 
PDF
Quick Wins
HighLoad2009
 
PPT
01 oracle architecture
Smitha Padmanabhan
 
PDF
Performance Scenario: Diagnosing and resolving sudden slow down on two node RAC
Kristofferson A
 
PDF
Spark Autotuning - Spark Summit East 2017
Alpine Data
 
Oracle Database In-Memory Option in Action
Tanel Poder
 
Dev nexus 2017
Roy Russo
 
Devnexus 2018
Roy Russo
 
Apache Spark At Scale in the Cloud
Rose Toomey
 
Apache Spark At Scale in the Cloud
Databricks
 
Spark on YARN
Adarsh Pannu
 
backgroundcommunicationandwaitevents-180124221026.pdf
ssuser785ce21
 
NoSQL with MySQL
FromDual GmbH
 
Brief Introduction to Parallella
Somnath Mazumdar
 
504_senegacnik_oracle_database_mutexes.pdf
cookie1969
 
My First 100 days with an Exadata (WP)
Gustavo Rene Antunez
 
Real World Performance - Data Warehouses
Connor McDonald
 
Performance Benchmarking: Tips, Tricks, and Lessons Learned
Tim Callaghan
 
Tos tutorial
manikainth
 
The power of linux advanced tracer [POUG18]
Mahmoud Hatem
 
OOW09 Ebs Tuning Final
jucaab
 
Quick Wins
HighLoad2009
 
01 oracle architecture
Smitha Padmanabhan
 
Performance Scenario: Diagnosing and resolving sudden slow down on two node RAC
Kristofferson A
 
Spark Autotuning - Spark Summit East 2017
Alpine Data
 

More from Enkitec (20)

PDF
Using Angular JS in APEX
Enkitec
 
PDF
Controlling execution plans 2014
Enkitec
 
PDF
Engineered Systems: Environment-as-a-Service Demonstration
Enkitec
 
PDF
Profiling Oracle with GDB
Enkitec
 
PDF
Oracle Performance Tools of the Trade
Enkitec
 
PDF
Oracle Performance Tuning Fundamentals
Enkitec
 
PDF
SQL Tuning Tools of the Trade
Enkitec
 
PDF
Using SQL Plan Management (SPM) to Balance Plan Flexibility and Plan Stability
Enkitec
 
PDF
Oracle GoldenGate Architecture Performance
Enkitec
 
PDF
OGG Architecture Performance
Enkitec
 
PDF
APEX Security Primer
Enkitec
 
PDF
How Many Ways Can I Manage Oracle GoldenGate?
Enkitec
 
PDF
Understanding how is that adaptive cursor sharing (acs) produces multiple opt...
Enkitec
 
PDF
Sql tuning made easier with sqltxplain (sqlt)
Enkitec
 
PDF
Profiling the logwriter and database writer
Enkitec
 
PDF
Fatkulin hotsos 2014
Enkitec
 
PDF
Combining ACS Flexibility with SPM Stability
Enkitec
 
PDF
Why You May Not Need Offloading
Enkitec
 
PDF
LOBS, BLOBS, CLOBS: Dealing with Attachments in APEX
Enkitec
 
PDF
Creating a Business Oriented UI in APEX
Enkitec
 
Using Angular JS in APEX
Enkitec
 
Controlling execution plans 2014
Enkitec
 
Engineered Systems: Environment-as-a-Service Demonstration
Enkitec
 
Profiling Oracle with GDB
Enkitec
 
Oracle Performance Tools of the Trade
Enkitec
 
Oracle Performance Tuning Fundamentals
Enkitec
 
SQL Tuning Tools of the Trade
Enkitec
 
Using SQL Plan Management (SPM) to Balance Plan Flexibility and Plan Stability
Enkitec
 
Oracle GoldenGate Architecture Performance
Enkitec
 
OGG Architecture Performance
Enkitec
 
APEX Security Primer
Enkitec
 
How Many Ways Can I Manage Oracle GoldenGate?
Enkitec
 
Understanding how is that adaptive cursor sharing (acs) produces multiple opt...
Enkitec
 
Sql tuning made easier with sqltxplain (sqlt)
Enkitec
 
Profiling the logwriter and database writer
Enkitec
 
Fatkulin hotsos 2014
Enkitec
 
Combining ACS Flexibility with SPM Stability
Enkitec
 
Why You May Not Need Offloading
Enkitec
 
LOBS, BLOBS, CLOBS: Dealing with Attachments in APEX
Enkitec
 
Creating a Business Oriented UI in APEX
Enkitec
 

Recently uploaded (20)

PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PPTX
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
defencerabbit Team
 
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
PDF
GamePlan Trading System Review: Professional Trader's Honest Take
SOFTTECHHUB
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PDF
Advanced IT Governance
University of Hertfordshire
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PDF
Modernizing your data center with Dell and AMD
Principled Technologies
 
Cloud computing and distributed systems.
kkkkkhan
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
defencerabbit Team
 
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
GamePlan Trading System Review: Professional Trader's Honest Take
SOFTTECHHUB
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
Advanced IT Governance
University of Hertfordshire
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Modernizing your data center with Dell and AMD
Principled Technologies
 

Drilling Deep Into Exadata Performance

  • 1. Drilling  Deep  into  Exadata  Performance  with  ASH,   SQL  Monitoring  and  ExaSnapper   Tanel  Põder   Enkitec  Europe     h.p://www.enkitec.com   h.p://blog.tanelpoder.com   www.enkitec.com     1      
  • 2. Intro:  About  me   •  Tanel  Põder   •  Former  Oracle  Database  Performance  geek   •  Present  Exadata  Performance  geek  ;-­‐)   •  My  Exadata  experience   •  I  have  had  the  luck  to  work  with  all   possible  Exadata  configuraMons  out  there   •  Exadata  V1  …  X2-­‐8   •  MulM-­‐rack  Exadatas  …   •  Even  a  mixed  rack  Exadata  (V2  <-­‐>  X2-­‐2  :)   Expert  Oracle  Exadata   book   (with  Kerry  Osborne  and   Randy  Johnson  of  Enkitec)   www.enkitec.com     2      
  • 3. Agenda   1.  A  li.le  detour  1:  Finding  non-­‐Exadata  friendly  SQL   2.  A  li.le  detour  2:  selecMvely  forcing  full  table  scans  in  mixed   workload  environments!   3.  A  li.le  bit  of  Smart  Scan  internals   4.  Reading  a  SQL  Monitoring  report  on  Exadata   •  …  and  where  it  falls  short   5.  Using  advanced  Exadata  performance  metrics   •  Exadata  Snapper  (ExaSnap)!!!   www.enkitec.com     3      
  • 4. Detour  1     Finding  non-­‐Exadata-­‐friendly  SQL   www.enkitec.com     4      
  • 5. 1)  Finding  top  non-­‐Exadata-­‐friendly  SQLs   •  OpMon  1:   •  Find  SQLs  which  wait  for  non-­‐smart  scan  IO  operaMons  the  most   •  ASH!   •  OpMon  2:   •  Find  SQLs  doing  the  most  disk  reads  without  Smart  Scans   •  Highest  MB  read  or  highest  IOPS   a)  ASH!   –  SUM(DELTA_READ_IO_REQUESTS)   –  SUM(DELTA_READ_IO_BYTES)   b)  ..or  join  to  V$SQLSTAT  (or  DBA_HIST_SQLSTAT)   –  SUM(PHYSICAL_READ_REQUESTS_DELTA)   –  SUM(PHYSICAL_READ_BYTES_DELTA)   www.enkitec.com     5      
  • 6. 2)  Are  these  long-­‐running  or  frequently  executed  short   queries?   •  Exadata  Smart  scans  are  not  opMmized  for  ultra-­‐frequent  execuMon   •  Smart  Scanning  1000s  of  Mmes  per  second  isn’t  good   •  That’s  why  Smart  Scans  are  not  good  for  your  OLTP-­‐queries   •  Here’s  an  analogy:   1.  Want  to  deliver  a  single  parcel  to  a  desMnaMon   •  Use  a  Ferrari   2.  Want  to  deliver  10  000  parcels  to  a  desMnaMon   •  Use  a  truck   3.   Want  to  deliver  10  000  000  parcels  to  a  desMnaMon   •  Use  a  freight  train   •  Smart  Scan  is  the  Exadata’s  freight  train   •  Brute  force  scanning  with  relaMvely  high  inerMa  to  get  started,  not  a  few   quick  (buffered)  I/O  operaMons  done  with  surgical  precision   www.enkitec.com     6      
  • 7. Demo  –  exafriendly.sql   •  Drill  down  into  ASH  wait  data:   SQL> @exadata/exafriendly.sql gv$active_session_history SESSION WAIT_CLASS EVENT SECONDS PCT ------- -------------------- ---------------------------------------- ---------- ------ ON CPU 192356 34.6 WAITING User I/O cell single block physical read 191838 34.5 WAITING User I/O db file parallel read 40577 7.3 WAITING User I/O cell smart table scan 28593 5.1 WAITING User I/O cell multiblock physical read 19424 3.5 WAITING User I/O direct path read temp 18398 3.3 WAITING Application enq: RO - fast object reuse 8690 1.6 WAITING User I/O direct path read 5812 1.0 ... PLAN_LINE USERNAME EVENT SECONDS PCT ------------------------------------- ------------ ---------------------------------- --------- ------ TABLE ACCESS BY LOCAL INDEX ROWID USER_104 cell single block physical read 40954 21.3 TABLE ACCESS BY INDEX ROWID USER_779 cell single block physical read 32129 16.7 INDEX RANGE SCAN USER_104 cell single block physical read 25272 13.2 TABLE ACCESS STORAGE FULL USER_49 cell single block physical read 9258 4.8 INDEX RANGE SCAN USER_779 cell single block physical read 4118 2.1 TABLE ACCESS STORAGE FULL USER_783 cell single block physical read 3641 1.9 UPDATE USER_104 cell single block physical read 3509 1.8 TABLE ACCESS BY INDEX ROWID USER_420 cell single block physical read 3341 1.7 MERGE USER_962 cell single block physical read 2910 1.5 www.enkitec.com     7      
  • 8. Demo  –  mon_topsql.sql   •  TOP  Time-­‐consuming  SQLs  with  IO  &  execuMon  counts   •  That  way  we’ll  separate  the  Ferraris  from  Freight  Trains!   •  The  “Ferraris”  aren’t  Exadata  smart  scan  friendly   www.enkitec.com     8      
  • 9. Other  sources  for  overview  informaMon   •  ASH  reports  (based  on  ASH  data  which  I’ve  used)   •  EM  12c  ASH  analyMcs   www.enkitec.com     9      
  • 10. Detour  2         How  to  force  full  table  scans  for  reports?   …while  keeping  the  OLTP  workloads  free  to  use  indexes?     1)  Make  all  indexes  invisible   2)  Set  op<mizer_use_invisible_indexes  =  true  for  OLTP  sessions     •  Stay  tuned  for  my  whitepaper  “SelecMve  Index  Visibility  for  Oracle  Exadata”     on  this  topic  :-­‐)   www.enkitec.com     10      
  • 11. Exadata  Architecture   •  All  DB  nodes  talk  to  all  (configured)  cells  -­‐  ASM  striped  data   •  A  cell  never  talks  to  another  cell  !!!   DB  Node  1   DB  Node  2   DB  Node  3   DB  Node  4   Cell  1   Cell  2   Cell  3   Cell  4   Cell  5   Cell  6   Cell  7   www.enkitec.com     11      
  • 12. Smart  Scans:  Asynchronous,  independent  prefetching   PARSING IN CURSOR #47233445473208 len=38 dep=0 uid=93 oct=3 lid=93 tim=1303079537221426 select * from t3 where owner like 'S%' END OF STMT PARSE #47233445473208:c=1000,e=8964,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=4161002650,ti EXEC #47233445473208:c=0,e=21,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4161002650,tim=1303 WAIT #47233445473208: nam='SQL*Net message to client' ela= 4 driver id=1413697536 #bytes WAIT #47233445473208: nam='SQL*Net more data to client' ela= 16 driver id=1413697536 #by WAIT #47233445473208: nam='reliable message' ela= 1337 channel context=11888341784 chann WAIT #47233445473208: nam='enq: KO - fast object checkpoint' ela= 143 name|mode=12634685 WAIT #47233445473208: nam='enq: KO - fast object checkpoint' ela= 130 name|mode=12634685 WAIT #47176529789912: nam='cell smart table scan' ela= 25 cellhash#=3176594409 p2=0 p3=0 WAIT #47176529789912: nam='cell smart table scan' ela= 882 cellhash#=2133459483 p2=0 p3= WAIT #47176529789912: nam='cell smart table scan' ela= 34 cellhash#=3176594409 p2=0 p3=0 WAIT #47176529789912: nam='cell smart table scan' ela= 837 cellhash#=2133459483 p2=0 p3= WAIT #47176529789912: nam='cell smart table scan' ela= 26 cellhash#=2133459483 p2=0 p3=0 WAIT #47176529789912: nam='cell smart table scan' ela= 22 cellhash#=379339958 p2=0 p3=0 The  waits  are  so  short  due  to  the   The  object  checkpoint-­‐ If  cell  smart  table/ asynchronous  nature  of  smart   related  wait  events  reliable   index  scan  waits   scans.  Cellsrvs  in  each  cell  process   message  and  enq:  KO  –  fast   show  up,  then  a   the  blocks  independently  to  fill   object  checkpoint  always   smart  scan  is   their  send  buffers  and  the  DB  just   precede  every  direct  path   a.empted   pulls  the  results  from  there   scan  (thus  smart  scan  too)   www.enkitec.com     12      
  • 13. Storage  cells  are  “shared  nothing”   •  And  they  don’t  see  what’s  happening  in  the  database  layer…   Database  Layer   A  large  table   Smart  Scan   Cell  Layer   Cell  1   Cell  2   Cell  3   Cell  4   Cell  5   Cell  6   Cell  7   www.enkitec.com     13      
  • 14. Physical  disks  -­‐  Simple  math   •  SequenMal  “brute  force”  scan  rate  150  MB/sec  per  disk   or     •  200  random  IOPS  per  disk   I’m  leaving   flash  cache   •  12  disks  in  a  storage  cell   out  for   simplicity  for   •  14  storage  cells  in  a  full  rack   now   •  150  *  12  *  14  =  25200  MB/sec  physical  disk  scanning  rate   •  If  doing  only  sequenMal  brute  force  scanning     However,   Index  scans   •  200  *  12  *  14  *  8kB  =  262.5  MB/sec   can  read   only  a  subset   •  Random  physical  read  rate  (index  scan!)  around  100x  slower!   of  data   www.enkitec.com     14      
  • 15. The  MoMvaMon  for  WriMng  Exadata  Snapper   [celladmin@enkcel01 ~]$ cellcli CellCLI: Release 11.2.2.4.0 - Production on Mon Mar 05 08:19:41 CST 2012 Copyright (c) 2007, 2011, Oracle. All rights reserved. Cell Efficiency Ratio: 699 CellCLI> ????   Following  just  the  Cell  Efficiency  RaVo  can  be  as   misleading  as  tuning  by  Buffer  Cache  Hit  RaVo!   ?   www.enkitec.com     15      
  • 16. Processing   Data  ReducMon,  IO  Avoidance,  Early  Filtering   Stage   SorMng,   aggregaMon,   joins  etc   Smart  Scan  returned  MB   +  Extra  block  reads   Temp  IO   Bugs,  chained   Smart  Scan  returned  MB   +  Extra  block  reads   rows,  migrated   rows,  read   Smart  Scan  returned  MB   consistency   Spinning  Disk  Read  MB   Flash  Reads   Storage   indexes   “Disk”  Read  MB   Reads  avoided                          Compressed  Data   Uncompressed  Data   Data  Volume   www.enkitec.com     16      
  • 17. NegaMve  Cell  Offload  Efficiency  raMo?   •  Must  understand  where  does  this  raMo  come  from   •  RaMo  of  which  Exact  metrics?   •  And  use  those  metrics  in  the  future   www.enkitec.com     17      
  • 18. NegaMve  Cell  Offload  Efficiency:  Data  Load  Example   Why  6  GB?   2  +  2  =  4  GB   The  “Bytes  returned  by  Exadata”   So,  write  IOs  also  influence  cell  offload   metric  actually  uses  the  “cell  physical   efficiency  calculaMon  (data  loads,  sort/join/ IO  interconnect  bytes”  metric   aggregate  operaMon  TEMP  usage).     internally,  which  includes  all  traffic,  not   Write  IOs  are  double/triple  mirrored  by   just  rows  returned  from  smart  scan.     ASM!   www.enkitec.com     18      
  • 19. InterpreMng  the  raw  metrics  with  Exadata  Snapper   •  IO  Efficiency  breakdown   •  How  much  physical  IO  did  we  really  do?   •  How  much  reads  /  how  much  writes?   •  How  much  disk  IO  did  we  avoid  doing  thanks  to  Flash  Cache?   •  How  much  disk  IO  did  we  avoid  doing  thanks  to  Storage  Indexes?   •  What  was  the  total  interconnect  traffic?   •  How  much  data  was  fully  processed  in  the  cell  (and  not  shipped  back   in  blocks  due  to  fallback  or  problems  in  the  cell?)   •  How  much  data  was  returned  by  a  Smart  Scan  (as  rows)?   •  @exadata/exasnap.sql   •  Beta  quality   •  Meant  to  complement  ASH  and  SQL  Monitoring  reports,  not  replace   them   www.enkitec.com     19      
  • 20. ExaSnap  example:  A  CTAS  statement   SQL> @exadata/exasnap % 123 124 ----------------------------------------------------------------------------------------------------------------- -- Exadata Snapper v0.5 BETA by Tanel Poder @ Enkitec - The Exadata Experts ( http://www.enkitec.com ) ----------------------------------------------------------------------------------------------------------------- SID CATEGORY METRIC IOEFF_PERCENTAGE MB ------ -------------------- ----------------------- ---------------------------------------------------- ---------- 1280 DB_IO DB_PHYSIO_MB |################################## | 4435 DB_IO DB_PHYSRD_MB |################# | 2219 DB_IO DB_PHYSWR_MB |################# The  real  disk  writes   | 2216 are  doubled  due  to   AVOID_DISK_IO PHYRD_FLASH_RD_MB | | 2 AVOID_DISK_IO PHYRD_STORIDX_SAVED_MB | ASM  double-­‐mirroring   | 0 DISK_IO SPIN_DISK_IO_MB |##################################################| 6649 DISK_IO SPIN_DISK_RD_MB |################# | 2217 DISK_IO SPIN_DISK_WR_MB |################################## | 4432 REDUCE_INTERCONNECT PRED_OFFLOAD_MB |################# | 2216 REDUCE_INTERCONNECT TOTAL_IC_MB |##################################################| 6444 REDUCE_INTERCONNECT SMART_SCAN_RET_MB |################ | 2009 REDUCE_INTERCONNECT NON_SMART_SCAN_MB |################################## | 4435 CELL_PROC_DEPTH CELL_PROC_DATA_MB |################# | 2232 CELL_PROC_DEPTH CELL_PROC_INDEX_MB | | 0 www.enkitec.com     20      
  • 21. ExaSnap  example  2:  Storage  Index  Savings   The  real  (spinning)  disk  reads  IO  was  only   1078  MB  thanks  to  1138  MB  of  disk  IO   SQL> @exadata/exasnap basic 90 91 avoided  due  to  storage  indexes:   ----------------------------------------------------------------------------------------------------------------- (2216  –  1138  =  1078)   -- Exadata Snapper v0.5 BETA by Tanel Poder @ Enkitec - The Exadata Experts ( http://www.enkitec.com ) ----------------------------------------------------------------------------------------------------------------- CATEGORY METRIC IOEFF_PERCENTAGE MB -------------------- ----------------------- ---------------------------------------------------- ---------- ---- DB_LAYER_IO DB_PHYSIO_MB |##################################################| 2216 DB_LAYER_IO DB_PHYSRD_MB |##################################################| 2216 DB_LAYER_IO DB_PHYSWR_MB | | 0 AVOID_DISK_IO PHYRD_FLASH_RD_MB | | 0 AVOID_DISK_IO PHYRD_STORIDX_SAVED_MB |########################## | 1138 REAL_DISK_IO SPIN_DISK_IO_MB |######################## | 1079 REAL_DISK_IO SPIN_DISK_RD_MB |######################## | 1078 REAL_DISK_IO SPIN_DISK_WR_MB | | 1 REDUCE_INTERCONNECT PRED_OFFLOADABLE_MB |##################################################| 2216 REDUCE_INTERCONNECT TOTAL_IC_MB | | 2 REDUCE_INTERCONNECT SMART_SCAN_RET_MB | | 2 REDUCE_INTERCONNECT NON_SMART_SCAN_MB | | 0 CELL_PROC_DEPTH CELL_PROC_DATA_MB |######################## | 1078 CELL_PROC_DEPTH CELL_PROC_INDEX_MB | | 0 All  1078  MB  worth  of  blocks  got  offloaded:   they  were  opened  and  processed  inside  the   cells  (data  layer)   www.enkitec.com     21      
  • 22. So,  why  isn’t  my  query  Exadata-­‐friendly?   www.enkitec.com     22      
  • 23. Demo  –  Drilling  down  into  a  SQL  execuMon   1.  SQL  Monitoring  report   •  ExecuMon  plan!   •  Where  is  most  of  the  response  Mme  spent  (retrieval  vs.  subsequent   processing)   •  Are  smart  scans  used  for  data  retrieval?   •  IO  MB  read  from  disks  vs  data  returned  from  the  cells   •  (also  called  the  offload  efficiency  raMo  but  knowing  the  underlying   numbers  is  way  be.er)     2.  ExaSnapper   •  Or  read  the  raw  v$sesstat  metrics  if  you  dare  ;-­‐)   www.enkitec.com     23      
  • 24. Warm-­‐up  case  study   •  TesMng  a•er  migraMon,  a  query  is  slow:   cell  single  block  physical  read   wait  events  in  a  parallel  query?   www.enkitec.com     24      
  • 25. Warm-­‐up  case  study:  check  the  execuMon  plan   •  Parallel  execuMon  plan  does  not  force  full  table  scans...   ------------------------------------------------------------------------------------------------------ | Id | Operation | Name | TQ |IN-OUT| PQ Distrib | ------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | | | | | 1 | PX COORDINATOR | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10002 | Q1,02 | P->S | QC (RAND) | |* 3 | FILTER | | Q1,02 | PCWC | | | 4 | HASH GROUP BY | | Q1,02 | PCWP | | | 5 | PX RECEIVE | | Q1,02 | PCWP | | | 6 | PX SEND HASH | :TQ10001 | Q1,01 | P->P | HASH | | 7 | NESTED LOOPS | | Q1,01 | PCWP | | | 8 | NESTED LOOPS | | Q1,01 | PCWP | Serial  stage   | | 9 | NESTED LOOPS | | Q1,01 | PCWP | in  a  parallel   | | 10 | BUFFER SORT | | Q1,01 | PCWC | plan   | | 11 | PX RECEIVE | | Q1,01 | PCWP | | | 12 | PX SEND ROUND-ROBIN | :TQ10000 | | S->P | RND-ROBIN | |* 13 | TABLE ACCESS BY GLOBAL INDEX ROWID| ORDERS | | | | |* 14 | INDEX RANGE SCAN | ORD_STATUS_IX | | | | | 15 | PARTITION HASH ITERATOR | | Q1,01 | PCWP | | Parallel  index   |* 16 | TABLE ACCESS BY LOCAL INDEX ROWID | CUSTOMERS | Q1,01 | PCWP | scans  (|on   |* 17 | INDEX UNIQUE SCAN | CUSTOMERS_PK | Q1,01 | PCWP | | | 18 | PARTITION HASH ITERATOR | | Q1,01 | PCWP | different   | |* 19 | INDEX RANGE SCAN | ORDER_ITEMS_PK | Q1,01 | PCWP | parMMons)   | | 20 | TABLE ACCESS BY LOCAL INDEX ROWID | ORDER_ITEMS | Q1,01 | PCWP | | ------------------------------------------------------------------------------------------------------ www.enkitec.com     25      
  • 26. Warm-­‐up  case  study  –  adjusted  execuMon  plan   •  A•er  forcing  full  table  scans:   ------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Pstart| Pstop | Fully  parallel   TQ |IN-OUT| PQ Distrib | execuMon   ------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | | plan   | | 1 | PX COORDINATOR | | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10003 | | | Q1,03 | P->S | QC (RAND) | |* 3 | FILTER | | | | Q1,03 | PCWC | | | 4 | HASH GROUP BY | | | | Q1,03 | PCWP | | | 5 | PX RECEIVE | | | | Q1,03 | PCWP | | | 6 | PX SEND HASH | :TQ10002 | | | Q1,02 | P->P | HASH | |* 7 | HASH JOIN | | | | Q1,02 | PCWP | | | 8 | PART JOIN FILTER CREATE | :BF0000 | | | Q1,02 | PCWP | | | 9 | PX RECEIVE | | | | Q1,02 | PCWP | | | 10 | PX SEND BROADCAST | :TQ10001 | | | Q1,01 | P->P | BROADCAST | |* 11 | HASH JOIN | | | | Q1,01 | PCWP | | | 12 | PART JOIN FILTER CREATE | :BF0001 | | | Q1,01 | PCWP | | | 13 | PX RECEIVE | | | | Q1,01 | PCWP | | | 14 | PX SEND BROADCAST | :TQ10000 | | | Q1,00 | P->P | BROADCAST | | 15 | PX BLOCK ITERATOR | | 1 | 16 | Q1,00 | PCWC | | |* 16 | TABLE ACCESS STORAGE FULL| ORDERS | 1 | 16 | Q1,00 | PCWP | | | 17 | PX BLOCK ITERATOR | |:BF0001|:BF0001| Q1,01 | PCWC | | |* 18 | TABLE ACCESS STORAGE FULL | CUSTOMERS |:BF0001|:BF0001| Q1,01 | PCWP | Full  table   | | 19 | PX BLOCK ITERATOR | |:BF0000|:BF0000| Q1,02 | PCWC | (parMMon)   | |* 20 | TABLE ACCESS STORAGE FULL | ORDER_ITEMS |:BF0000|:BF0000| Q1,02 | PCWP | | scans  with   ------------------------------------------------------------------------------------------------------------- 18 - storage(:Z>=:Z AND :Z<=:Z AND ("C"."NLS_TERRITORY"='AMERICA' AND 
 bloom  filter   SYS_OP_BLOOM_FILTER(:BF0000,"C"."CUSTOMER_ID"))) offloading   www.enkitec.com     26      
  • 27. Case  2:  Response  Mme  24  seconds  –  where  is  it  spent?   1)  Full  scan:  TABLE  ACCESS  STORAGE  FULL   Smart  scan   will  help!   2)  But  waiMng  for  a  buffered  read  wait  event   •  cell  mul<block  physical  read  instead  of  cell  smart  table/index  scan   www.enkitec.com     27      
  • 28. Case  2:  The  same  query  runs  in  3  seconds  with  Smart  Scan   •  So,  why  is  it  faster?   •  Data  retrieval  (ACCESS)  from  storage  is  faster   www.enkitec.com     28      
  • 29. Case  2:  Now  let’s  do  something  with  the  data…   •  Same  query,  with  an  extra  GROUP  BY:   •  Takes  3.8  minutes!  (228  seconds)   •  Why?  Let’s  see  what’s  taking  the  extra  Mme:   www.enkitec.com     29      
  • 30. Checkpoint   •  Smart  Scans  make  the  data  retrieval  from  storage  faster   •  Any  other  problems  require  the  usual  SQL  &  DB  opMmizaMon   www.enkitec.com     30      
  • 31. Smart  Scans  do  not  make  any  of  these  things  faster  (by  design):   •  Index  unique/range/skip  scans  *     *  A  prerequisite  for   smart  scans  is  a  full   •  SorMng   segment  scan  anyway   •  AggregaMon  (GROUP  BY)   •  AnalyMc  FuncMons   •  Filters  that  can’t  be  pushed  down  to  table  level:   •  WHERE  t1.col  +  t2.col  =  1   •  Any  funcMon  calls  in  projecMon  (select  list)   •  PL/SQL  funcMon  calls  in  projecMon  (or  WHERE  clause)   •  Nested  loop  joins,  sort-­‐merge  joins  and  FILTER  lookups   •  Hash  joins  are  special  though   •  So,  you’ll  have  to  see  where  is  your  bo.leneck!   This  is  not  a  full   •  SQL  Monitoring  report  is  a  good  start   list  of  limitaMons.   www.enkitec.com     31      
  • 32. A  query  bo.lenecked  by  data  processing,  not  retrieval   •  A  SQL  performing  data  load  and  spills  to  TEMP   www.enkitec.com     32      
  • 33. Case  3:  Case  insensiMve  search   ALTER SESSION SET nls_comp = LINGUISTIC; The  plan  hash  value  does  not  change  as  it  doesn’t   ALTER SESSION SET nls_sort = BINARY_CI; take  the  predicate  differences  into  account!   SELECT SUM(credit_limit) FROM soe.customers SELECT SUM(credit_limit) FROM soe.customers WHERE cust_first_name LIKE 'j%' WHERE cust_first_name LIKE 'j%' Plan hash value: 296924608 Plan hash value: 296924608 FAST ~ 2 seconds SLOW ~14 seconds ----------------------------------------------- ----------------------------------------------- | Id | Operation | Name | | Id | Operation | Name | ----------------------------------------------- ----------------------------------------------- | 0 | SELECT STATEMENT | | | 0 | SELECT STATEMENT | | | 1 | SORT AGGREGATE | | | 1 | SORT AGGREGATE | | |* 2 | TABLE ACCESS STORAGE FULL| CUSTOMERS | |* 2 | TABLE ACCESS STORAGE FULL| CUSTOMERS| ----------------------------------------------- ----------------------------------------------- Predicate Information Predicate Information --------------------------------------------- ---------------------------------------------- 2 - storage("CUST_FIRST_NAME" LIKE 'j%') 2 - filter(NLSSORT filter("CUST_FIRST_NAME" LIKE 'j%') ("OWNER",'nls_sort=''BINARY_CI''') >HEXTORAW('7300’) ) Where’s  the   storage  predicate?   www.enkitec.com     33      
  • 34. FAST  –  2  seconds   Case3:   SLOW  –  14  seconds   www.enkitec.com     34      
  • 35. Thank  you  –  and  oh,  wait!   •  Advanced  Exadata  Performance  seminar!   •  By  Tanel  Poder   •  SystemaMc  Exadata  Performance  TroubleshooMng  and  OpMmizaMon   •  3-­‐day  seminar  available  from  November   •  We’ll  go  very  deep!     •  QuesMons:  tanel@tanelpoder.com   •  Blog:    h.p://blog.tanelpoder.com   •  ConsulMng:  h.p://www.enkitec.com   www.enkitec.com     35