DataEngConf SF16 - Data Asserts: Defensive Data Science
3 likes1,093 views
1) Complex data pipelines can introduce bugs that compound as dependencies increase. Engineers manage complexity through encapsulation, clear APIs, and integration tests. 2) Data scientists require semantic correctness but making assumptions introduces risks. Sanity checks on fields like verifying formats and constraints help identify potential errors. 3) Defensive data science through data asserts maintains quality by clearly defining trust boundaries and assumptions. Checks should match expectations and be revisited regularly as upstream changes can impact pipelines.
![How do Data Scientists identify potential
errors?
Some follow-on fact is absurd…
… which leads to investigation …
… which finds a broader problem
If [potential conclusion], then we must have 3 billion
OneDrive users…
… because my user table doesn’t have a primary key …
… so I should aggregate by user.](https://image.slidesharecdn.com/dataengconfdataasserts1-160414233203/85/DataEngConf-SF16-Data-Asserts-Defensive-Data-Science-10-320.jpg)
![What are your Assumptions?
If I conclude “Users who upload files to OneDrive are XXX% more likely
to buy Office if they also sent mail through Mobile Outlook”, I’m
making many silent assumptions:
Field Assumptions
User Id • Logged and PII-encrypted similarly in Outlook and OneDrive
• Correctly logging timestamp for Office purchase
• User Id isn’t empty or missing
OneDrive activity • Wasn’t automated traffic [identified by a certain flag].
Email Activity • Mobile client identifiers are correct.
All • Any upstream changes to OneDrive, Office, or Exchange
data have been communicated to pipeline owners.](https://image.slidesharecdn.com/dataengconfdataasserts1-160414233203/85/DataEngConf-SF16-Data-Asserts-Defensive-Data-Science-11-320.jpg)
DataEngConf SF16 - Data Asserts: Defensive Data Science
- 1. Data Asserts Defensive Data Science Tommy Guy Microsoft
- 2. Observation: Complexity In Pipeline
- 3. Our pipeline: DATA!!! Insight! Direction! Strategy!
- 4. Our pipeline in reality: bugs tend to compound DATA!!!
- 5. How do Engineers Manage Complexity? Encapsulate: create functions/classes/subsystems with clear APIs. This helps isolate complexity Integration Tests: ensure that the components interact correctly. This helps identify breaking changes.
- 6. Data introduces a few complications Pipelines take many upstream dependencies Researcher use cases are frequently unknown and unanticipated by data providers. Pushing requirements upstream to all producers is Sisyphean.
- 7. We are not talking about data pipeline tests The data pipeline teams: Are all rows that are produced stored • Counter fields to ensure no dropped rows • Sentinel events to measure join fidelity Are availability SLAs being met? • Progressive server-client merging
- 8. Data Scientists Require Semantic Correctness Does this field mean what I think it does?
- 9. How do Data Scientists identify potential errors?
- 10. How do Data Scientists identify potential errors? Some follow-on fact is absurd… … which leads to investigation … … which finds a broader problem If [potential conclusion], then we must have 3 billion OneDrive users… … because my user table doesn’t have a primary key … … so I should aggregate by user.
- 11. What are your Assumptions? If I conclude “Users who upload files to OneDrive are XXX% more likely to buy Office if they also sent mail through Mobile Outlook”, I’m making many silent assumptions: Field Assumptions User Id • Logged and PII-encrypted similarly in Outlook and OneDrive • Correctly logging timestamp for Office purchase • User Id isn’t empty or missing OneDrive activity • Wasn’t automated traffic [identified by a certain flag]. Email Activity • Mobile client identifiers are correct. All • Any upstream changes to OneDrive, Office, or Exchange data have been communicated to pipeline owners.
- 12. What are your Sanity Checks? • If a column “OfficeId” is really a user id, it has certain known properties: • Observation: these sorts of checks take place when the pipeline is set up, but they may not be re-checked very often. Assumption Why does it matter? Never null/empty Causes job-breaking data skew issues Users are 1:* with Tenants Logical constraint: sign you are missing something. Very high cardinality If this isn’t true, it’s unlikely that it’s a user-id. All rows in event data join to it Otherwise, your data is incomplete. Matches a certain regex Sanity check: if this isn’t true, it’s unlikely that it’s a user-id.
- 13. Data Asserts: Defensive Data Science
- 14. Data Asserts: Maintain Quality
- 15. Data Asserts: Clear Trust Boundaries
- 16. These should match! Data Asserts: Defensive Data Science
- 17. Data Asserts in Production: A few Observations • Most of the analysis-impacting assertion failures we’ve seen were actually errors in our assumptions not errors in the pipeline. • Good tests beget good code: we’ve had to modularize our code in order to produce testable chunks that get re-used in pipelines. • Data Asserts is the backbone to data provenance. A data conclusion can directly link all of the assumptions about the input that we made.