Bi-temporal rdbms 2014

Editor's Notes

• #7: With versioning: We track all changes to the data, who made the change and when ― not just the last change made ― without using cumbersome “history” tables. Examining the history is simply a matter of writing the right query. We may “look back in time” to see how the data looked on a particular date (transaction time) ― even if some of the entities have subsequently been deleted. This is particularly useful in auditing and report generation. We may “look back in time” to see how the data should have looked on a particular date (valid time). It becomes possible to implement an “undo” feature in the database. So a change to the data, no matter how extensive, may be reversed or rolled all the way back to the way it was at a particular time. Data may be “pre-inserted” or “pre-updated” to take effect at a particular date and time in the future. The new or modified data will remain invisible to queries until the effective date and time is reached. We don’t have to track history through extraordinary means (such as separate “history” tables). Every versioned entity will automatically maintain a complete history, from creation to deletion with no additional effort on the part of the applications. Not only are changes tracked, but also when each change was made or became effective. Thus we can also track the length of time each version was in effect. All referential integrity constraints to and from versioned tables work the same way as with non-versioned tables. Queries against versioned tables (while we realize will be a bit more complicated) follow a consistent pattern and, once learned, are the same no matter the table or the database in which it is used. Versioning takes place at the field level. A “versioned table” is just a table that has had some of its fields versioned, not necessarily all. The underlying table can be altered (fields added or dropped) without impacting the versioned fields — unless, of course, a versioned field is dropped. Versioned fields may be unversioned and unversioned fields may be versioned with little impact. Applications like CVS and SVN ― which allow us to look back in time to follow the changes made to source code ― may easily be developed to give us the same ability to track changes to our data. All this in a transactional database without maintaining separate snapshot or audit tables, and with only minor impact on performance.
• #10: Versioning State Of The Art I only recently became aware of the following synopsis of database versioning found in Wikipedia. The entry is titled Slowly Changing Dimension (SCD) which gives a brief description of the problem of tracking changes in information. This, of course, applies mainly to data warehousing applications, but still provides excellent background as to the nature of the solutions in use today. It lists six methods that are, I assume, currently used to handle modifications to data, from doing nothing (Type 1) to a hodge-podge design that uses a date to designate when the data became effective, a date to designate when it is no longer effective, an indicator to signify the row that is now current and a sequence value (Type 6/Hybrid). There are several versions of Type 6, one that seems as though it would almost be workable. There are several distinct disadvantages to even the best of these methods. There are two dates used: start and end. The start date indicates when the data in the row becomes effective. The end date indicates when the data in the row is no longer effective. The problem with this is that we now have to keep adjacent versions in sync with each other. This creates the problem I refer to as a “row spanning dependency”: when data in one row of a table is dependent on data in another row of the same table. When we need to write a new version that is to take effect immediately, we first have to update the existing current version to store “now” in the end date field and write the new version with “now” in the start date field. So all changes to the data involves an Update/Insert pair of operations (sometimes referred to as an “Upsert”). There is also a problem if we want to delete the current version, in effect “undo” the last change. The current version may be deleted but the last version must now be located to return the end date to the value understood to mean “current”. Referential integrity is still not possible as the version timestamp or a sequence value must be part of the primary key or unique index. Any foreign key in another table referring to the versioned table can only refer to one version. Actually, one of the methods solved the referential integrity problem in a clever way. The primary key consists of the natural key and a sequence number starting at 0. Any foreign key reference would refer to this row. When the second version is created, the first row is copied back into the table, becoming sequence number 1 and the first row is actually updated with the new data. So the current version is always sequence number 0 and previous versions take on higher sequence values. This also means that sequential versions must be kept synchronized and all changes involve a two-step, Insert/Update, operation. There are also the complicated (and unmentioned in the article) problem of maintaining sequential values for each entity in the table. All current versions of all entities have value 0 so this column can't be an Identity (or autogenerated) field.
• #11: Basic Requirements Of Maintaining Temporally Sensitive Data To make working with temporally sensitive data relatively simple and less error prone, we must demand certain minimum requirements. We must have a date and time value to signify when the version takes effect. But we should not have to specify when the version is no longer effective. Let's just agree that a version remains in effect until superseded by the next version or the entity is deleted. This eliminates row-spanning dependencies (the necessity of keeping different rows of the table in sync with each other). We should not have a separate field used to indicate the current version. Whether it be a 'Y/N' indication or a set numeric value, this is just another area where errors can creep in. (What happens when we have more than one version with a 'Y' in this field? Don't even try to argue that it would never happen!) When a new version is written to the table, no changes at all should be needed to any previous version including the current version. A logical Update of entity data is converted into a physical Insert, but this should be the only operation needed and it cannot be allowed to introduce systemic errors into the data. We should be able to create foreign key references to temporally sensitive entities no differently than to “normal” entities. The versioning method should be usable for any and all temporally sensitive data, whatever the entity and wherever in the enterprise it is needed. Easier said than done? Possibly, but we have to start somewhere and I contend that if we don't meet those basic requirements, we don't have a workable solution. The first two rules of versioning already mentioned (no overlapping and no gaps), are brought about by requirements 1 and 2 above. With only one date, the effective date, there is no way for the versions to get out of sync. The effective date of one version is the “end” date of the previous version. It is not possible, therefore, for overlapping of versions or gaps between versions. By using only the effective dates to determine which is the current version (or the version in effect at any particular moment in time) we eliminate any confusion or errors possible with other indicators. And because the effective date becomes part of the primary key of the versioned data, it is not possible to have two versions with the same effective date. By using only the effective date to determine the one version of interest, we satisfy requirement 3. When we modify the versioned entity ― which is performed by inserting a new version ― no changes need be made to any data already existing in the entity table.
• #12: Requirement 4 was the tough one. But the solution turned to be simplicity itself. It took me an entire month to come up with a workable solution but it took another three months before I saw just how simple it turned out to be. I had arrived at an answer by such a convoluted path that I kept seeing it as complicated ― workable but complicated ― until one day I drew it all out on a whiteboard to explain it to a coworker. By ignoring the path to the solution and just looking at the solution itself, the simplicity just leaps out at you. This simplicity also affects its universal applicability, thus requirement 5 is met. The solution turns out to be nothing more than to consider versioning as just another form of relational normalization ― a Version Normal Form (vnf), if you will. I doubt any experienced data modeler actually steps through the normalization process as we are taught in Data Management 101 class in college. We very quickly reach the point where we think in 3nf (or BCNF) and our design hits the paper already in the required normal form. But let's consider the process for a moment. We have identified all the attributes we will maintain for an entity. We have identified all the candidate keys and selected one to be the Primary Key (or elected to use a surrogate). Now we perform the first step in the normalization process (1nf) and as a result, we now have two tables ― the primary entity table (or independent table) and, say, one additional table (dependent table) which contains repeating attributes. Have we created a problem with referential integrity? Of course not. In fact, data integrity is maintained within the entity by using a foreign key in the dependent tables back to the independent table. Any references to the entity from other entities will be resolved by referencing the independent table. It has the one entity primary key and the normalization process will not effect that. The dependent tables will have a primary key of its own, usually consisting of the entity key field and an identifying additional value or values. The same thing applies when we continue the normalization process. At the end, we end up with one logical entity that may be spread across several physical tables, but there is one table that is the “entity” table (Employees, Customers, Parts, Classes, Items, etc.) and it is this table that is the target of any foreign key that refers to the entity. There is no problem with having both normalization and referential integrity. Thus, after placing the entity in temporal normal form, we get all the benefits of versioning and referential integrity is unaffected. We also get the benefit of having versioning in a pattern (normalization) that database programmers are intimately familiar with. So the learning curve of working with versioned data is pretty close to zero.
• #19: Now when Sally notifies us of her name change, we just take the current versioned row from Employee­Versions, change “Jones” to “Smith” and Effective to the date of the wedding and insert the new record. When we query the database, we see “Smith.” But if we query asking “show me the data for employee 1001 as of Dec. 31, 2005,” (or any date before then) we will get “Jones.” Because this pattern is bi-temporal, we can also ask “show me the data for employee 1001 as it existed in the database on Jan, 10, 2006 and we will get “Jones” because, on that date, the database had not yet been updated with the correct name.
• #21: Now when Sally notifies us of her name change, we just take the current versioned row from Employee­Versions, change “Jones” to “Smith” and Effective to the date of the wedding and insert the new record. When we query the database, we see “Smith.” But if we query asking “show me the data for employee 1001 as of Dec. 31, 2005,” (or any date before then) we will get “Jones.” Because this pattern is bi-temporal, we can also ask “show me the data for employee 1001 as it existed in the database on Jan, 10, 2006 and we will get “Jones” because, on that date, the database had not yet been updated with the correct name.
• #23: Now when Sally notifies us of her name change, we just take the current versioned row from Employee­Versions, change “Jones” to “Smith” and Effective to the date of the wedding and insert the new record. When we query the database, we see “Smith.” But if we query asking “show me the data for employee 1001 as of Dec. 31, 2005,” (or any date before then) we will get “Jones.” Because this pattern is bi-temporal, we can also ask “show me the data for employee 1001 as it existed in the database on Jan, 10, 2006 and we will get “Jones” because, on that date, the database had not yet been updated with the correct name.
• #47: Now when Sally notifies us of her name change, we just take the current versioned row from Employee­Versions, change “Jones” to “Smith” and Effective to the date of the wedding and insert the new record. When we query the database, we see “Smith.” But if we query asking “show me the data for employee 1001 as of Dec. 31, 2005,” (or any date before then) we will get “Jones.” Because this pattern is bi-temporal, we can also ask “show me the data for employee 1001 as it existed in the database on Jan, 10, 2006 and we will get “Jones” because, on that date, the database had not yet been updated with the correct name.