Tracking your data across the fourth dimension
0 likes321 views
The document discusses temporal databases and how they handle data involving time. It explains the key temporal aspects of decision time, valid time, and transaction time. Decision time records when a decision was made, valid time defines the time period when a fact was true, and transaction time tracks when a fact was considered true in the database. The document also provides examples of implementing these temporal aspects and discusses challenges around temporal primary keys, foreign keys, and querying temporal data. Finally, it discusses current database vendor support for the SQL:2011 standard for representing temporal data.
Tracking your data across the fourth dimension
- 1. TRACKING YOUR DATA ACROSS THE FOURTH DIMENSION Jeremy Cook CodeDaze 2016
- 2. ““A temporal database is a database with built-in support for handling data involving time… –Wikipedia
- 3. DON’T DATABASES ALREADY HANDLE TIME BASED DATA?
- 4. DATABASES ARE GOOD AT ‘NOW’ ➤ Create ➤ Read ➤ Update ➤ Delete ➤ At any point we only see the current state of the data
- 6. ““Tonight @JCook21 explained temporal databases and I’m sure my brain is now leaking out of my nose… – Jeff Carouth, https://twitter.com/ jcarouth/status/496842218674470912
- 7. TEMPORAL ASPECTS ➤ Term to describe different types of ‘temporality’ ➤ Current theory defines three aspects ➤ We use temporal aspects all the time, representing them in a DB is hard…
- 9. DECISION TIME ➤ Records the time at which a decision was made ➤ Modelled as a single value ➤ Allows for granularity through the data type used
- 10. DECISION TIME EmpId Name Hire Date Decision to Hire 1 Jeremy 2014-03-03 2014-01-20 2 PJ 2015-01-02 2013-12-15 3 Johnny 2013-08-20 2013-08-20
- 11. VALID TIME
- 12. ““In temporal databases, valid time (VT) is the time period during which a database fact is valid in the modelled reality. -Wikipedia
- 13. VALID TIME ➤ Modelled as a range between two points in time ➤ Lower bound is always closed but upper bound can be open ➤ Also known as Application Time
- 14. VALID TIME EmpId Name Position StartVT EndVT 1 Jeremy Dev 2014-03-03 2015-01-19 1 Jeremy Senior Dev 2015-01-20 ∞ 2 PJ Dev 2015-01-02 2016-01-30 2 PJ Manager 2016-01-31 ∞ 3 Johnny Director 2013-08-20 2016-09-30 3 Johnny Vice President 2016-10-01 ∞
- 15. JOB DONE?
- 16. VALID-TIME ON ITS OWN MAY NOT BE ENOUGH… Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Planet Billions of years ago ∞
- 17. VALID-TIME ON ITS OWN MAY NOT BE ENOUGH… Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Dwarf planet Billions of years ago ∞
- 18. VALID-TIME ON ITS OWN MAY NOT BE ENOUGH… Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Plutoid Billions of years ago ∞
- 19. VALID-TIME ON ITS OWN MAY NOT BE ENOUGH… Name Type StartVT EndVT Saturn Planet Billions of years ago ∞ Pluto Planet Billions of years ago 2006 Pluto Dwarf planet 2006 2008 Pluto Plutoid 2008 ∞
- 20. TRANSACTION TIME
- 21. ““In temporal databases, transaction time (TT) is the time period during which a fact stored in the database is considered to be true.” - Wikipedia
- 22. TRANSACTION TIME ➤ Modelled as a range between two points in time ➤ Lower bound is always closed but upper bound can be open ➤ Also known as System Time
- 23. TRANSACTION TIME Name Type StartVT EndVT StartTT EndTT Pluto Planet Billions of years ago ∞ 1930 2006 Pluto Dwarf planet Billions of years ago ∞ 2006 2008 Pluto Plutoid Billions of years ago ∞ 2008 ∞
- 24. HOW MANY TEMPORAL ASPECTS SHOULD YOU USE? ➤ As many or few as your application needs! ➤ Tables that implement two aspects are bi-temporal ➤ You can implement more aspects, in which case you have multi temporal tables
- 25. IS YOUR HEAD SPINNING? ➤ Decision time records when a decision was taken ➤ Valid Time records the period of time for which the fact is valid in the modelled reality ➤ Transaction Time records the period of time for which the fact is considered to be true in the modelled reality
- 27. PERIOD DATA TYPE ➤ Table component, capturing a pair of columns defining a start and end date ➤ Not a new data type, but metadata about columns in the table ➤ Closed-open constraint ➤ Enforces that end time > start time
- 28. VALID TIME ➤ Also called application time in SQL:2011 ➤ Modelled as a pair of date or timestamp columns with a period ➤ Name of the columns and period is up to you
- 29. TEMPORAL PRIMARY KEYS ➤ SQL:2011 allows a valid time period to be named as part of a primary key ➤ Without this how can you trace the evolution of data? ➤ Can also enforce that the valid time periods do not overlap
- 30. TEMPORAL FOREIGN KEYS ➤ What happens if a parent and child table both define valid time periods? ➤ It doesn’t make sense to allow a row in a child table to reference a row in a parent table where the valid time does not overlap ➤ SQL:2011 allows valid time periods to be part of foreign key constraints ➤ For the constraint to be considered satisfied the valid time period in the child table must be contained within the valid time period of one or more contiguous rows in the parent table
- 31. QUERYING VALID TIME TABLES ➤ Can query against valid time columns as normal - they’re just normal table columns ➤ When querying the database you need to specify what valid time period you’re interested in ➤ Updates and deletes can be performed for a period of a valid time time period ➤ Can create periods in queries and use new predicates to query for valid time
- 32. TRANSACTION TIME ➤ Also known as system time in SQL:2011 ➤ Modelled as two DATE or TIMESTAMP columns ➤ New predicates to query across transaction time periods. ➤ Management of the columns for the period is handled by the database for you
- 33. TRANSACTION TIME ➤ Because the system manages transaction time: ➤ Not possible to alter transaction time values in the past ➤ Not possible to add future dated transaction time values
- 35. CURRENT SUPPORT ➤ IBM DB2 ➤ Very good support ➤ Oracle 12c ➤ Valid time and transaction time support ➤ SQL Server 2016 ➤ Only supports system/transaction time ➤ PostgreSQL ➤ 9.1 and earlier: temporal contributed package ➤ 9.2 native ranged data types ➤ Handful of others implemented as extensions
- 36. HOW DO I ADD THIS STUFF TO MY CURRENT SCHEMA?
- 37. SOME BIG PROBLEMS ➤ Temporal primary keys ➤ Temporal foreign keys ➤ Where are the boundaries of your temporal model? ➤ Making changes in a timeline is hard… ➤ Coaching users about the kind of changes they're making ➤ Querying for the data is harder
- 38. IMPLEMENTING VALID TIME - ONE TABLE ➤ Add a pair of date time columns to your table for the valid time period. ➤ Can make these part of your primary key ➤ Good: ➤ Simpler model ➤ Ugly: ➤ Foreign keys become hard
- 39. IMPLEMENTING VALID TIME - TWO TABLES ➤ ‘Main’ table contains data that is valid now ➤ Second table contains data for other valid time periods ➤ Process to ‘promote’ data as it becomes valid ➤ Good: ➤ Easy to retrofit to an existing schema ➤ Ugly: ➤ More processing/logic ➤ What level of granularity can you achieve and accept with promoting data?
- 40. IMPLEMENTING TRANSACTION TIME ➤ Add a column recording transaction time start to your table ➤ For each table create a backup table mirroring the columns in the main table, adding a transaction time end column too ➤ Create a trigger that fires on each update or delete to copy old values from the main table to the backup table ➤ Should add transaction time end to the backup table ➤ Should also update the transaction time start to now in the main table if the operation is an update
- 41. IMPLEMENTING TRANSACTION TIME ➤ Things to consider: ➤ Extra complexity ➤ How long should backup data be kept for? ➤ Do you optimize for fast reads or writes?
- 42. MORE INFORMATION ➤ Wikipedia article on Temporal Databases ➤ Temporal features in SQL:2011 (PDF)
- 43. THANKS FOR LISTENING! ➤ Any questions? ➤ Contact me: ➤ @JCook21 ➤ jeremycook0@icloud.com