SAP Hana transaction log reading

Recently there have been some SAP notes about Hana transaction log readers provided by other companies. As I was hired to build a solution for one vendor, I thought providing technical insights might be a good idea for users to better evaluate the claims of these software vendors and SAP's statements. No details can be provided, NDA.

The technical details about transaction log reading in case of Hana, what other databases offer and CDC tool requirements can be found here: https://rtdi.io/transaction-log-reader-for-sap-hana/

Back to the claims...

SAP's statements are:

• We do not provide an API for transaction log reading
• Hence we do not offer a certification for this
• The transaction log format can change at any time
• We do not see any value

In contrast, vendors motivate:

• We are SAP certified
• Reading the transaction log is the most efficient method of change data capture for batch and realtime
• It is very easy to use and error free
• Can handle all cases and all Hana deployment options

Regarding certification

The situation is quite obvious when reading one level deeper. There are certifications for Hana, e.g. reading data via JDBC. As all tools do that, they can be certified. For this use case and only this use case. There is no API for transaction log reading and SAP certifies the correctness of partners interacting via APIs only, hence there can be no certification.

But why are there transaction log reader solutions? Because it is the most effective way to get change data. Customers know that, they ask for it. So instead of blaming others, SAP should provide an API and a certification.

The SAP Hana team is using transaction log reading for the system table replication feature already, they just do not want to open up to others, not even other SAP teams. That is the reason why SLT is trigger based, SAP SDI is trigger based, SAP Datasphere is using the S/4 embedded SLT - which is trigger based.

Triggers work fairly well for the normal case, but if you want to annoy SAP during a PoC, execute a single update/delete statement changing millions or billions of rows in the source. This is not uncommon, e.g. when archiving data or during migrations/upgrades. A SQL statement that runs within seconds thanks to the compactness of the Hana transaction log suddenly takes hours when the SLT triggers are present.

This also puts the other SAP statements in perspective. As SAP Hana is using the transaction log themselves across database versions, the format will not change. And even if, that is why software gets tested. Why SAP does not feel the need for a transaction log reader API beats me. The impact of triggers on the source database performance is in the 10% range, especially the way SLT was built. Is it because SAP does not feel the pressure because it is not paid by SAP? The customer must increase the hardware resources and pay more for the Hana license? It might not be a conscious decision but it is the outcome.

Regarding simplicity

If testing a log reader solution, the things I would look at carefully are

• All data types including rarely used ones and extreme values, e.g. dates before 1400 to validate the correct calendar type is used
• The output format created, especially full width and before image values
• Correct handling of upsert statements, truncate table, alter table partitions as these are special in the transaction log
• Savepoint handling
• Schema evolution, meaning adding a column. Especially for a restart of the transaction log reader during the time the table got changed
• Encrypted log files, encrypted backup files, changing the Hana encryption keys
• Hana as a cluster
• Row tables as these are different from column tables in the transaction log