MERGE SQL Statement: Lesser Known Facets
- 1. blog.sqlora.com@Andrej_SQL MERGE SQL Statement: Lesser Known Facets Andrej Pashchenko
- 2. About me • Working at Trivadis Germany, Düsseldorf • Focusing on Oracle: • Data Warehousing • Application Development • Application Performance • Course instructor „Oracle New Features for Developers“ @Andrej_SQL blog.sqlora.com
- 5. • MERGE is a part of SQL 2003 and has been introduced in Oracle 9i • Since then, the MERGE has been well adopted and widely used, but sometimes still has some confusing or unexpected behavior
- 6. ORA-30926
- 7. ORA-30926 • ORA-30926 is definitely the most confusing error related to MERGE • The error description is somewhat confusing too: • One of the reasons is clearly documented:
- 8. • Create an „overlaping“ bonus list and try to merge it in employee table INSERT INTO scott.bonus (ename, job, sal, comm) SELECT ename, job, sal, sal*0.2 comm FROM scott.emp WHERE deptno = 30; INSERT INTO scott.bonus (ename, job, sal, comm) SELECT ename, job, sal, sal*0.1 comm FROM scott.emp WHERE job = 'MANAGER’; SQL> MERGE INTO scott.emp e 2 USING scott.bonus b 3 ON (b.ename = e.ename) 4 WHEN MATCHED THEN UPDATE set e.comm = b.comm; USING scott.bonus b * ERROR at line 2: ORA-30926: unable to get a stable set of rows in the source tables ORA-30926 - Example Sales department 20% Each manager 10% But SALES also has a manager: BLAKE will be updated twice
- 9. • Check the duplicates in the source with respect to the ON-keys: SQL> MERGE INTO scott.emp e 2 USING scott.bonus b 3 ON (b.ename = e.ename) 4 WHEN MATCHED THEN 5 UPDATE SET e.comm = b.comm; USING scott.bonus b * ERROR at line 2: ORA-30926: unable to get a stable set of rows in the source tables ORA-30926 - How to find the problem? SQL> SELECT ename 2 FROM scott.bonus b 3 GROUP BY ename 4 HAVING COUNT(*) > 1; ENAME ---------- BLAKE SQL>SELECT ename, MAX(comm) comm FROM scott.bonus b GROUP BY ename; • Find the correct way to avoid duplicates. Often this is a business question, e.g. use MAX or SUM:
- 10. • Fix the problem in the source data or directly in your query: SQL> MERGE INTO scott.emp e 2 USING (SELECT ename, MAX(comm) comm 3 FROM scott.bonus b 4 GROUP BY ename) b 5 ON (b.ename = e.ename) 6 WHEN MATCHED THEN UPDATE set e.comm = b.comm; 8 rows merged. ORA-30926 Fixing the problem • What does the documentation say about ORA-30926: ORA-30926: unable to get a stable set of rows in the source tables Cause: A stable set of rows could not be got because of large dml activity or a non- deterministic where clause. Action: Remove any non-deterministic where clauses and reissue the dml.
- 11. The whole execution three times? • Have you noticed that the execution takes much longer if you get ORA-30926? SQL> MERGE INTO scott.emp e 2 USING scott.bonus b 3 ON (b.ename = e.ename) 4 WHEN MATCHED THEN UPDATE SET e.comm = b.comm; USING scott.bonus b * ERROR at line 2: ORA-30926: unable to get a stable set of rows in the source tables ----------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | ----------------------------------------------------------------- | 0 | MERGE STATEMENT | | 3 | | 0 | | 1 | MERGE | EMP | 3 | | 0 | | 2 | VIEW | | 3 | | 21 | |* 3 | HASH JOIN | | 3 | 9 | 21 | | 4 | TABLE ACCESS FULL| BONUS | 3 | 9 | 27 | | 5 | TABLE ACCESS FULL| EMP | 3 | 14 | 26 | -----------------------------------------------------------------
- 12. Write Consistency and DML restarts
- 13. Write Consistency and DML Restarts (UPDATE) Session 1 Session 2 SQL> UPDATE emp 2 SET sal = 1000 3 WHERE ename = 'JAMES'; t1 SQL> UPDATE emp 2 SET comm = nvl(comm,0) + 1000 3 WHERE sal < 1000; t2 COMMIT;t3 t4 SQL> SELECT ename, sal, comm ENAME SAL COMM FROM emp WHERE sal < 1000; ---------- ---------- ---------- SMITH 800 JAMES 950 ENAME SAL COMM ---------- ---------- ---------- SMITH 800 1000 JAMES 1000 Cannot update JAMES‘s row an waits Session 2 can now update JAMES, but JAMES‘s salary is not less than 1000 anymore
- 14. Write Consistency and DML Restart Get SCN1 Identify rows to be updated in consistent mode (per SCN1) Get row in current mode tracked columns unchanged? Rollback all changes made so far Request new SCN SELECT FOR UPDATE (current mode) Identify rows to be updated in consistent mode (new SCN) tracked columns unchanged? Update locked rows Update the row Yes YesNo Up to 5000 times No Tracked: columns in WHERE and :OLD, :NEW values in BEFORE EACH ROW trigger
- 15. Write Consistency and DML Restarts (MERGE) Session 1 Session 2 SQL> UPDATE emp 2 SET sal = 1000 3 WHERE ename = 'JAMES'; t1 SQL> MERGE INTO emp t 2 USING (SELECT 1000 comm FROM dual) q 3 ON (t.sal < 1000) 4 WHEN MATCHED THEN UPDATE 5 SET t.comm = nvl(t.comm,0)+q.comm; t2 COMMIT;t3 t4 ENAME SAL COMM ---------- ---------- ---------- SMITH 800 1000 JAMES 1000 1000 Waits for locked row Session 2 can now update JAMES, but JAMES‘s salary is not less than 1000 anymore
- 16. Write Consistency and DML Restarts (MERGE) Session 1 Session 2 SQL> UPDATE emp 2 SET sal = 1000 3 WHERE ename = 'JAMES'; t1 SQL> MERGE INTO emp t 2 USING (SELECT 1000 comm FROM dual) q 3 ON (t.sal < 1000) 4 WHEN MATCHED THEN UPDATE 5 SET t.comm = nvl(t.comm,0)+q.comm 6 WHERE (t.sal < 1000) ; t2 COMMIT;t3 t4 ENAME SAL COMM ---------- ---------- ---------- SMITH 800 1000 JAMES 1000 Waits for locked row
- 17. Write Consistency and DML Restarts (MERGE) Session 1 Session 2 SQL> UPDATE emp 2 SET comm = 500 3 WHERE ename = 'JAMES'; t1 SQL> MERGE INTO emp t 2 USING (SELECT 1000 comm FROM dual) q 3 ON (t.sal < 1000) 4 WHEN MATCHED THEN UPDATE 5 SET t.comm = nvl(t.comm,0)+ q.comm; t2 COMMIT;t3 t4 ENAME SAL COMM ---------- ---------- ---------- SMITH 800 1000 JAMES 950 1500 Waits for locked row
- 18. Write Consistency and DML Restart • MERGE can show a different behavior regarding DML restarts • There was a bug until 18c about not tracking ON-columns, but it is still there even in 19c in some cases • But MERGE is tracking columns in SET clause thus preventing “lost updates” during running statements (no replace for locking strategy in your app) • In case of DML restart triggers can fire multiple times, so avoid any non- transactional logic or autonomous transactions in triggers!
- 20. Write Consistency and DML Restart • Obviously Oracle is using the same mechanism of mini rollbacks as with write consistency also to ensure its deterministic behavior • SET-columns are tracked • Even within the same session updating the column to another value will be detected • The MERGE statement will be restarted • As a result, we can observe the mentioned triple effort: MERGE and than rollback, SELECT FOR UPDATE and then MERGE again
- 21. ORA-38104
- 22. ORA-38104 • Oracle doesn’t allow to update columns used in ON clause • If you feel like you have to do this, verify your requirements carefully • Sometimes useful for ad hoc data manipulation, fixing erroneous data, etc. EMPNO ENAME 7839 KING 7782 CLARK 7934 MILLER EMPNO ROLE_NAME 7839 DEFAULT 7782 SALES EMP EMP_ROLES • Assign the role ACCOUNTING to each employee of deptno=10 • If an employee is assigned the DEFAULT role, overwrite this assignment. EMPNO ROLE_NAME 7839 ACCOUNTING 7782 SALES 7782 ACCOUNTING 7934 ACCOUNTING EMP_ROLES MERGE?
- 23. ORA-38104 • The straightforward approach doesn’t work: SQL> MERGE INTO emp_roles t 2 USING (SELECT empno, 'ACCOUNTING' role_name, 'DEFAULT' old_role 3 FROM emp 4 WHERE deptno = 10 5 ) q 6 ON (t.empno = q.empno and t.role_name = q.old_role) 7 WHEN MATCHED THEN UPDATE SET t.role_name = q.role_name 8 WHEN NOT MATCHED THEN INSERT VALUES (q.empno, q.role_name) ; ON (t.empno = q.empno and t.role_name = q.old_role) * ERROR at line 6: ORA-38104: Columns referenced in the ON Clause cannot be updated: "T"."ROLE_NAME"
- 24. ORA-38104 • Using WHERE clause instead of ON only seems to work, because the result is wrong: no new role assignment for MILLER (7782) EMPNO ROLE_NAME 7839 ACCOUNTING 7782 SALES 7934 ACCOUNTING EMP_ROLES SQL> MERGE INTO emp_roles t 2 USING (SELECT empno, 'ACCOUNTING' role_name 3 FROM emp 4 WHERE deptno =10 5 ) q 6 ON (t.empno = q.empno) 7 WHEN MATCHED THEN UPDATE 8 SET role_name = q.role_name 9 WHERE t.role_name = 'DEFAULT' 10 WHEN NOT MATCHED THEN INSERT (empno, role_name) 11 VALUES (q.empno, q.role_name) ; 2 rows merged.
- 25. ORA-38104 – Using ROWID • Doing the whole logic in USING subquery and merging on ROWID • At the price of increased complexity and performance penalty of one more join EMPNO ROLE_NAME 7839 ACCOUNTING 7782 SALES 7782 ACCOUNTING 7934 ACCOUNTING EMP_ROLES SQL> MERGE INTO emp_roles t 2 USING (SELECT r.rowid rid, 'ACCOUNTING' new_role_name 3 , e.empno 4 FROM emp e LEFT JOIN emp_roles r 5 ON e.empno = r.empno 6 AND r.role_name = 'DEFAULT' 7 WHERE e.deptno = 10 8 ) q 9 ON (t.rowid = q.rid ) 10 WHEN MATCHED THEN UPDATE 11 SET role_name = q.new_role_name 12 WHEN NOT MATCHED THEN INSERT (empno, role_name) 13 VALUES (q.empno, q.new_role_name) ; 3 rows merged.
- 26. ORA-38104 – Fooling the Parser • Using a subquery EMPNO ROLE_NAME 7839 ACCOUNTING 7782 SALES 7782 ACCOUNTING 7934 ACCOUNTING EMP_ROLES SQL> MERGE INTO emp_roles t 2 USING (SELECT empno, 'ACCOUNTING' role_name 3 , 'DEFAULT' old_role 4 FROM emp 5 WHERE deptno = 10 6 ) q 7 ON ( t.empno = q.empno 8 AND (SELECT t.role_name FROM dual) = q.old_role ) 9 WHEN MATCHED THEN UPDATE SET role_name = q.role_name 10 WHEN NOT MATCHED THEN INSERT (empno, role_name) 11 VALUES (q.empno, q.role_name) ; 3 rows merged. Idee: Blog Lukas Eder
- 27. ORA-38104 – Fooling the Parser • Using a view as a merge target and hiding a column inside NVL() EMPNO ROLE_NAME 7839 ACCOUNTING 7782 SALES 7782 ACCOUNTING 7934 ACCOUNTING EMP_ROLES SQL> MERGE INTO 2 (SELECT empno, role_name 3 , nvl(role_name,'NEVER') check_role_name 4 FROM emp_roles) t 5 USING (SELECT empno, 'ACCOUNTING' role_name 6 , 'DEFAULT' old_role 7 FROM emp 8 WHERE deptno = 10 ) q 9 ON ( t.empno = q.empno 10 AND t.check_role_name = q.old_role ) 11 WHEN MATCHED THEN UPDATE SET role_name = q.role_name 12 WHEN NOT MATCHED THEN INSERT (empno, role_name) 13 VALUES (q.empno, q.role_name) ; 3 rows merged. Idee: Blog Lukas Eder
- 28. ORA-38104 – Fooling the Parser • Using row value expressions EMPNO ROLE_NAME 7839 ACCOUNTING 7782 SALES 7782 ACCOUNTING 7934 ACCOUNTING EMP_ROLES SQL> MERGE INTO emp_roles t 2 USING (SELECT empno, 'ACCOUNTING' role_name 3 , 'DEFAULT' old_role 4 FROM emp 5 WHERE deptno = 10 ) q 6 ON ( 1=2 OR 7 (t.empno, t.role_name) = ((q.empno, q.old_role)) ) 8 WHEN MATCHED THEN UPDATE SET role_name = q.role_name 9 WHEN NOT MATCHED THEN INSERT (empno, role_name) 10 VALUES (q.empno, q.role_name) ; 3 rows merged. Idee: Blog Lukas Eder
- 29. ORA-38104 – Can be executed multiple times • All previous examples could only be executed once! This one is “multi-executable” EMPNO ROLE_NAME 7839 ACCOUNTING 7782 SALES 7782 ACCOUNTING 7934 ACCOUNTING EMP_ROLES SQL> MERGE INTO emp_roles t 2 USING ( 3 SELECT DISTINCT 4 FIRST_VALUE(r.rowid) 5 OVER(PARTITION BY e.empno 6 ORDER BY DECODE(r.role_name, 7 'ACCOUNTING',1,2)) rid 8 , e.empno, 'ACCOUNTING' new_role_name 9 FROM emp e LEFT JOIN emp_roles r 10 ON e.empno = r.empno 11 AND r.role_name in ('DEFAULT', 'ACCOUNTING') 12 WHERE e.deptno = 10 13 ) q 14 ON (t.rowid = q.rid ) 15 WHEN MATCHED THEN UPDATE SET role_name = q.new_role_name 16 WHEN NOT MATCHED THEN INSERT (empno, role_name) 17 VALUES (q.empno, q.new_role_name) ; 3 rows merged.
- 31. Direct Path and MERGE? SQL> MERGE /*+ append */ 2 INTO tx2 USING tx ON (tx.n = tx2.n) 3 WHEN NOT MATCHED THEN INSERT (N) VALUES (tx.n) ; 10,000 rows merged. ---------------------------------------- | Id | Operation | Name | ---------------------------------------- | 0 | MERGE STATEMENT | | | 1 | MERGE | TX2 | | 2 | VIEW | | |* 3 | HASH JOIN RIGHT OUTER| | | 4 | TABLE ACCESS FULL | TX2 | | 5 | TABLE ACCESS FULL | TX | ---------------------------------------- SQL> SELECT count(*) FROM tx2; ORA-12838: cannot read/modify an object after modifying it in parallel SQL> INSERT /*+ append*/ INTO tx2 2 SELECT n FROM tx 3 WHERE n NOT IN (SELECT n FROM tx2); 10,000 rows inserted. ----------------------------------------------- | Id | Operation | Name ----------------------------------------------- | 0 | INSERT STATEMENT | | 1 | LOAD AS SELECT | TX2 | 2 | OPTIMIZER STATISTICS GATHERING | |* 3 | HASH JOIN RIGHT ANTI NA | | 4 | TABLE ACCESS FULL | TX2 | 5 | TABLE ACCESS FULL | TX ----------------------------------------------- direct path write has happend
- 32. Space Management with PDML and MERGE? SQL> MERGE /*+ append parallel*/ 2 INTO t_tgt_join t0 3 USING ( SELECT ... ---------------------------------------- | Id | Operation | ---------------------------------------- | 0 | MERGE STATEMENT | | 1 | PX COORDINATOR | | 2 | PX SEND QC (RANDOM) | | 3 | MERGE | | 4 | PX RECEIVE | SEGMENT_NAME BLOCKS CNT ------------ ------ ------- T_TGT_JOIN 8 1088 ... 7 rows ... T_TGT_JOIN 128 4647 ... 20 rows ... T_TGT_JOIN 1024 34 30 rows selected. SQL> INSERT /*+ append parallel */ 2 INTO t_tgt_join t0 3 SELECT ... -------------------------------------------------- |Id | Operation -------------------------------------------------- | 0 | INSERT STATEMENT | 1 | PX COORDINATOR | 2 | PX SEND QC (RANDOM) | 3 | LOAD AS SELECT (HIGH WATER MARK BROKERED) | 4 | OPTIMIZER STATISTICS GATHERING SEGMENT_NAME BLOCKS CNT ------------ ---------- --------- T_TGT_JOIN 8 1024 T_TGT_JOIN 128 4216 T_TGT_JOIN 256 20 T_TGT_JOIN 384 2 T_TGT_JOIN 512 8 T_TGT_JOIN 640 8 T_TGT_JOIN 768 4 T_TGT_JOIN 896 1 T_TGT_JOIN 1024 134
- 33. Conclusion • It‘s easy to find out the reason for ORA-30926 • But be careful when fixing it only by technical methods. Another discussion with business stakeholders may be necessary. • Don’t execute heavy batch DML in environments with large user activity. • Don‘t use any non-transactional logic inside the triggers • Be careful overcoming ORA-38104 restriction. Rethink your use case if possible. • Keep in mind a different behavior for PDML and Online Statistics Gathering
- 34. Links • Oracle documentation, MERGE • Tom Kyte about DML restarts I and II • Ruslan Dautkhanov, Oracle’s Write Consistency • Lukas Eder about ORA-38104 • MERGE and ORA-30926