Assuring spatial and temporal data integrity via constraints and triggers in PostgreSQL
1 like946 views
The document discusses the use of PostgreSQL constraints and triggers to ensure the integrity of spatial and temporal data in urban logistics applications. It contrasts traditional data collection methods with modern techniques leveraging smartphones and machine learning to gather travel behavior data more efficiently. Additionally, it outlines potential pitfalls in data integrity and provides solutions through the implementation of constraints and triggers to maintain correct datasets.
![New ways to collect same data
Use end user devices (mostly smartphones), to (legally) collect
how people move (GPS points).
Old way
– Ask people to fill up forms
New way
– Ask people to look at their GPS points and fill up forms(?)
New way 2.0
– Use [buzzwords] machine learning / AI to avoid asking people
to fill up forms
4](https://image.slidesharecdn.com/pugconstraintstriggers-161025132133/85/Assuring-spatial-and-temporal-data-integrity-via-constraints-and-triggers-in-PostgreSQL-12-320.jpg)
![New ways to collect same data
Use end user devices (mostly smartphones), to (legally) collect
how people move (GPS points).
Old way
– Ask people to fill up forms
New way
– Ask people to look at their GPS points and fill up forms(?)
New way 2.0
– Use [buzzwords] machine learning / AI to avoid asking people
to fill up forms
– But we need data to train the algorithms
4](https://image.slidesharecdn.com/pugconstraintstriggers-161025132133/85/Assuring-spatial-and-temporal-data-integrity-via-constraints-and-triggers-in-PostgreSQL-13-320.jpg)
![New ways to collect same data
Use end user devices (mostly smartphones), to (legally) collect
how people move (GPS points).
Old way
– Ask people to fill up forms
New way
– Ask people to look at their GPS points and fill up forms(?)
New way 2.0
– Use [buzzwords] machine learning / AI to avoid asking people
to fill up forms
– But we need data to train the algorithms
– Also, algorithms will give wrong predictions
4](https://image.slidesharecdn.com/pugconstraintstriggers-161025132133/85/Assuring-spatial-and-temporal-data-integrity-via-constraints-and-triggers-in-PostgreSQL-14-320.jpg)
![New ways to collect same data
Use end user devices (mostly smartphones), to (legally) collect
how people move (GPS points).
Old way
– Ask people to fill up forms
New way
– Ask people to look at their GPS points and fill up forms(?)
New way 2.0
– Use [buzzwords] machine learning / AI to avoid asking people
to fill up forms
– But we need data to train the algorithms
– Also, algorithms will give wrong predictions
– How about a continuous feedback loop between users and ML
algorithms
4](https://image.slidesharecdn.com/pugconstraintstriggers-161025132133/85/Assuring-spatial-and-temporal-data-integrity-via-constraints-and-triggers-in-PostgreSQL-15-320.jpg)
![New ways to collect same data
Use end user devices (mostly smartphones), to (legally) collect
how people move (GPS points).
Old way
– Ask people to fill up forms
New way
– Ask people to look at their GPS points and fill up forms(?)
New way 2.0
– Use [buzzwords] machine learning / AI to avoid asking people
to fill up forms
– But we need data to train the algorithms
– Also, algorithms will give wrong predictions
– How about a continuous feedback loop between users and ML
algorithms
– What can go wrong?
4](https://image.slidesharecdn.com/pugconstraintstriggers-161025132133/85/Assuring-spatial-and-temporal-data-integrity-via-constraints-and-triggers-in-PostgreSQL-16-320.jpg)
![Dealing with predictions
Machine learning algorithms infer
– Time periods for trips (when people travel between
destinations)
– Time periods for triplegs (when people travel with different
travel modes [between destinations])
– Destination of trips (where people travel)
– Purpose of trips (why people travel)
– Travel modes (how people travel)
6](https://image.slidesharecdn.com/pugconstraintstriggers-161025132133/85/Assuring-spatial-and-temporal-data-integrity-via-constraints-and-triggers-in-PostgreSQL-20-320.jpg)
![Dealing with predictions
Machine learning algorithms infer
– Time periods for trips (when people travel between
destinations)
– Time periods for triplegs (when people travel with different
travel modes [between destinations])
– Destination of trips (where people travel)
– Purpose of trips (why people travel)
– Travel modes (how people travel)
Users can look at the output
– And confirm that it is correct
– Or modify it to bring it to the correct state
Alone
Using a weird UI
...
6](https://image.slidesharecdn.com/pugconstraintstriggers-161025132133/85/Assuring-spatial-and-temporal-data-integrity-via-constraints-and-triggers-in-PostgreSQL-21-320.jpg)
Assuring spatial and temporal data integrity via constraints and triggers in PostgreSQL
- 1. Assuring spatial and temporal data integrity via constraints and triggers in PostgreSQL Adrian C. Prelipcean acpr@kth.se adrianprelipceanc@gmail.com
- 2. Ego page Adrian C. Prelipcean 1. PhD student at KTH – Transportation systems and geoinformatics – Data collection and analysis methods for travel behaviour – Lab responsible for multiple courses GIS Algorithms (Java) Spatial Databases (PostgreSQL, PostGIS, pgRouting) Web and Mobile GIS (NodeJS, ExpressJS, Apache Cordova, PostgreSQL) 2. Tech responsible at Airmee – Airmee - a smart platform for urban logistics – Currently working on MVPs (from the backend to the frontend) – PostgreSQL as the backend 2
- 3. Background Travel behaviour - understand how people travel and make decisions We used to ask people how they are travelling - fill in paper forms, answer telephone interviews, fill in web forms. Found out how to adapt the infrastructure of our cities to handle a growing population. 3
- 4. Background Travel behaviour - understand how people travel and make decisions We used to ask people how they are travelling - fill in paper forms, answer telephone interviews, fill in web forms. Found out how to adapt the infrastructure of our cities to handle a growing population. However, also found out that: – Physical mail is not that popular. 3
- 5. Background Travel behaviour - understand how people travel and make decisions We used to ask people how they are travelling - fill in paper forms, answer telephone interviews, fill in web forms. Found out how to adapt the infrastructure of our cities to handle a growing population. However, also found out that: – Physical mail is not that popular. – Filling up forms is not really a hobby. 3
- 6. Background Travel behaviour - understand how people travel and make decisions We used to ask people how they are travelling - fill in paper forms, answer telephone interviews, fill in web forms. Found out how to adapt the infrastructure of our cities to handle a growing population. However, also found out that: – Physical mail is not that popular. – Filling up forms is not really a hobby. – Method doesn’t scale well. 3
- 7. New ways to collect same data Use end user devices (mostly smartphones), to (legally) collect how people move (GPS points). 4
- 8. New ways to collect same data Use end user devices (mostly smartphones), to (legally) collect how people move (GPS points). Old way – Ask people to fill up forms 4
- 9. New ways to collect same data Use end user devices (mostly smartphones), to (legally) collect how people move (GPS points). Old way – Ask people to fill up forms New way 4
- 10. New ways to collect same data Use end user devices (mostly smartphones), to (legally) collect how people move (GPS points). Old way – Ask people to fill up forms New way 4
- 11. New ways to collect same data Use end user devices (mostly smartphones), to (legally) collect how people move (GPS points). Old way – Ask people to fill up forms New way – Ask people to look at their GPS points and fill up forms(?) 4
- 12. New ways to collect same data Use end user devices (mostly smartphones), to (legally) collect how people move (GPS points). Old way – Ask people to fill up forms New way – Ask people to look at their GPS points and fill up forms(?) New way 2.0 – Use [buzzwords] machine learning / AI to avoid asking people to fill up forms 4
- 13. New ways to collect same data Use end user devices (mostly smartphones), to (legally) collect how people move (GPS points). Old way – Ask people to fill up forms New way – Ask people to look at their GPS points and fill up forms(?) New way 2.0 – Use [buzzwords] machine learning / AI to avoid asking people to fill up forms – But we need data to train the algorithms 4
- 14. New ways to collect same data Use end user devices (mostly smartphones), to (legally) collect how people move (GPS points). Old way – Ask people to fill up forms New way – Ask people to look at their GPS points and fill up forms(?) New way 2.0 – Use [buzzwords] machine learning / AI to avoid asking people to fill up forms – But we need data to train the algorithms – Also, algorithms will give wrong predictions 4
- 15. New ways to collect same data Use end user devices (mostly smartphones), to (legally) collect how people move (GPS points). Old way – Ask people to fill up forms New way – Ask people to look at their GPS points and fill up forms(?) New way 2.0 – Use [buzzwords] machine learning / AI to avoid asking people to fill up forms – But we need data to train the algorithms – Also, algorithms will give wrong predictions – How about a continuous feedback loop between users and ML algorithms 4
- 16. New ways to collect same data Use end user devices (mostly smartphones), to (legally) collect how people move (GPS points). Old way – Ask people to fill up forms New way – Ask people to look at their GPS points and fill up forms(?) New way 2.0 – Use [buzzwords] machine learning / AI to avoid asking people to fill up forms – But we need data to train the algorithms – Also, algorithms will give wrong predictions – How about a continuous feedback loop between users and ML algorithms – What can go wrong? 4
- 17. Data 5
- 18. Data 5
- 19. Data 5
- 20. Dealing with predictions Machine learning algorithms infer – Time periods for trips (when people travel between destinations) – Time periods for triplegs (when people travel with different travel modes [between destinations]) – Destination of trips (where people travel) – Purpose of trips (why people travel) – Travel modes (how people travel) 6
- 21. Dealing with predictions Machine learning algorithms infer – Time periods for trips (when people travel between destinations) – Time periods for triplegs (when people travel with different travel modes [between destinations]) – Destination of trips (where people travel) – Purpose of trips (why people travel) – Travel modes (how people travel) Users can look at the output – And confirm that it is correct – Or modify it to bring it to the correct state Alone Using a weird UI ... 6
- 22. What can go wrong? Invalid periods – e.g., traveled to work between 08:00 AM and 07:00 AM (same day) Overlapping periods – e.g., traveled to work between 08:00 AM and 09:00 AM, and traveled to the supermarket between 08:30 AM and 09:30 Discontinuous periods – e.g., traveled to work between 08:00 AM to 09:00 AM, worked between 09:00 AM and 11:30 AM worked, and traveled to a restaurant between 13:00 AM to 13:20 Improper nests – e.g., traveled to work between 08:00 AM to 09:00 AM, during which I walked between 08:20 and 09:00 7
- 23. How to prevent things from going wrong? 8
- 24. How to prevent things from going wrong? 8
- 25. How to prevent things from going wrong? 8
- 26. Avoiding headaches Constraints Good for enforcing conditions on the table’s attributes Specified at table declaration or afterward, via ALTER Types: – Check constraints (row-level), e.g., age >0 – Not null constraints (row-level), e.g., name not null – Unique constraints (table-level), e.g., two people cannot have the same ID – Primary key (table-level) - unique + not null – Foreign key (between tables) - referential integrity to other tables – Exclude constraints (table level) - unique + table-level check 9
- 27. Avoiding headaches Triggers Useful for enforcing constraints across multiple tables (works fine for same table constraints, as well) Can be used to implement complex logic Types – Event trigger Global, for the database Captures DDL events (CREATE, ALTER, DROP, COMMENT, GRANT, REVOKE) – Data change trigger Local, for the table Captures DML events (INSERT, UPDATE, TRUNCATE, DELETE) Can be run BEFORE or AFTER the data change event Can be conditioned by a WHEN clause Triggers can be written in any procedural language with event trigger support, or in C 10
- 28. Generate test data - Tables 11
- 29. Generate test data - Tables 11
- 30. Generate test data - Tables 11
- 31. Generate test data - Locations 12
- 32. Generate test data - Locations 12
- 33. Generate test data - Trips and triplegs 13
- 34. Generate test data - Trips and triplegs 13
- 35. Check Constraints ERROR: new row for relation "trips" violates check constraint "valid_periods" DETAIL: Failing row contains (5, 2016-10-25 07:37:28.6, 2016-10-25 06:37:28.6) 14
- 36. Not Null Constraints ERROR: null value in column "from_time" violates not-null constraint DETAIL: Failing row contains (6, null, null). 15
- 37. Primary Key Constraints ERROR: duplicate key value violates unique constraint "trips_pkey" DETAIL: Key (id)=(1) already exists. 16
- 38. Foreign Key Constraints ERROR: insert or update on table "triplegs" violates foreign key constraint "triplegs_trip_id_fkey" DETAIL: Key (trip_id)=(100) is not present in table "trips". 17
- 39. Exclusion Constraints ERROR: conflicting key value violates exclusion constraint "non_overlapping_trip_periods" DETAIL: Key (tsrange(from_time, to_time)) = (["2016-10-25 12:17:00","2016-10-25 12:20:00")) conflicts with existing key (tsrange(from_time, to_time)) = (["2016-10-25 12:00:00","2016-10-25 13:00:00")). 18
- 40. Triggers - definition example 19
- 41. Triggers - minimum two locations per tripleg ERROR: proposed period modification of tripleg 1 does not contain enough locations 20
- 42. Triggers - time update of trip affects neighbors 21
- 43. Triggers - time update of trip affects neighbors 21
- 44. Triggers - time update of trip affects neighbors 21
- 45. Triggers - time update of trip affects (nested) triplegs 22
- 46. Triggers - time update of trip affects (nested) triplegs 22
- 47. Triggers - time update of trip affects (nested) triplegs ERROR: proposed period modification of tripleg 4 does not contain enough locations Trigger attached to the Triplegs invalidated an operation on the Trips table and rolled back the transaction As long as the rules are properly defined and implemented, data integrity is maintained 22
- 48. Wrapping up Triggers and constraints Advantages – Can be used to implement your unified custom logic (write once, fix once) – Constraints are great for enforcing checks on a row-level or table-level (exclusion constraints), as well as referential integrity (foreign keys) – For multiple table logic (or single table logic), triggers can do almost anything - use responsibly Disadvantages – Documentation can be overwhelming – Cascading triggers can throw you in a loop (remove the WHEN conditions on trip_period_integrity trigger) – Performance overhead – Cascading triggers are difficult to debug and the actions are not always intuitive Advice – Write unit tests (pgTap) to validate that a changed / new trigger doesn’t break your functionality 23
- 49. References Docs: – Triggers - https://www.postgresql.org/docs/current/ static/sql-createtrigger.html – Constraints - https://www.postgresql.org/docs/ current/static/ddl-constraints.html – Exclusion constraints - https://www.postgresql.org/docs/current/static/ sql-createtable.html#SQL-CREATETABLE-EXCLUDE – Examples of exclusion constraints on ranges - https://www.postgresql.org/docs/current/static/ rangetypes.html#RANGETYPES-CONSTRAINT Other presentations on constraints and triggers – Jim Mlodgenski (more on triggers) - http://www.slideshare.net/jim_mlodgenski/ an-introduction-to-postresql-triggers – Robert Haas (triggers cheat sheet) - http://www. slideshare.net/pgconf/introduction-to-triggers – Magnus Hagander (exclusion constraints) - https://www.hagander.net/talks/BeyondUNIQUE.pdf 24
- 50. References (part 2) More on exclusion constraints – Depesz - https://www.depesz.com/2010/01/03/ waiting-for-8-5-exclusion-constraints/ – Davis Jeff - http://thoughts.davisjeff.com/2010/09/25/ exclusion-constraints-are-generalized-sql-unique/ Github examples: – For this talk - https://github.com/adrianprelipcean/PUG_Stockholm – For the tracking system mentioned at the beginning (database + mobile apps + annotation UI) - https://github.com/Badger-MEILI 25
- 51. Thank you for your attention! Questions and Discussions Adrian C. Prelipcean http://adrianprelipcean.github.io/ acpr@kth.se @Adi Prelipcean