OSDC 2019 | Automating Security in Your Data Pipline by Troy Harvey
0 likes139 views
The document discusses the automation of data pipeline security with an emphasis on privacy, outlining historical perspectives on privacy and the ethical responsibilities of developers. It highlights several regulations, including GDPR and CCPA, and introduces automation techniques for managing privacy through data processing, access management, and encryption strategies. The document also raises critical considerations for implementing security measures in data pipelines, emphasizing the need for a privacy-first approach.
OSDC 2019 | Automating Security in Your Data Pipline by Troy Harvey
- 1. Automating Data Pipeline Security
- 2. 1 2 4 3 5 6
- 5. Carta’s Data Team is Hiring 🎉
- 6. Automating Data Pipeline Security
- 7. Automating Data Pipeline Security Privacy
- 8. 3 Big Ideas 1. Privacy has a strange history. 2. Privacy-first systems are designed by people with a professional ethic. 3. Privacy can be automated away. Automating security in your data pipeline privacy
- 9. 1. Strange History of Privacy
- 16. 16  “The actio iniuriarum was, in Roman law, a delict which served to protect the non-patrimonial aspects of a person's existence – who a person is rather than what a person has.”
- 20. ©1979 "The Invention of the Right to Privacy" by Dorothy J. Glancy
- 23. Software is eating the world.
- 24. “Audit defensibility is too low a bar when it comes to our customer’s privacy.”
- 25. GDPR EU General Data Protection Regulation ● Right of access ● Pseudonymisation ● Right of erasure ● Records of processing activities ● Privacy by design CCPA California Consumer Privacy Act ● Know what personal information is being collected ● Right to erasure ● Know whether their personal information is being shared, and if so, with whom ● Opt-out of the sale of their personal information Privacy Regulation
- 28. “The security posture of your weakest vendor is the security posture of your entire organization.”
- 30. Blank Slide
- 32. ● Airflow DAGs to move data into S3 and Redshift ● DAG: Directed Acyclic Graph ● Operator/Task: A node in the graph ● Airflow runs dbt Workflow manager from Airbnb Apache Airflow
- 33. Apache Airflow ● Open source boilerplate for running Airflow in Docker ● Used at Carta Dockerized Airflow
- 35. How do we keep up with the sensitive columns being added in source data? Automating the blacklist updates Stale Blacklist
- 36. ● dbt tests fail when the result set is not empty. ● The records returned by dbt test are the offending records. Automated data tests dbt test
- 37. ● dbt tests fail when the result set is not empty. ● The records returned by dbt test are the offending records. Automated data tests dbt test
- 38. We have a custom access management system called Gatekeeper. Tools for requesting and granting access Automating Access
- 39. This example uses our IAM Service Account custom Terraform module to create a new Revenue Service account user with access to a single S3 data lake bucket. Automate Data Lake access Terraform Modules
- 40. Data Warehouse Migrations ● sql-migrate: Excellent cli and migrations library written in Go. ● Extended to support Jinja templating. We can rebuild the Warehouse from code.
- 41. Pseudonymity Disguised identity or “false name” ©2019 Alex Ewerlöf "GDPR pseudonymization techniques"
- 42. Pseudonymity: Obfuscation 👍 Easy to do in any language. 👍 No impact to downstream systems. 👎 Can be unscrambled. Scrambling or mixing up data
- 43. Pseudonymity: Masking 👍 Simple. 👍 Owner can verify the last 4 digits. 👎 Some pieces of the real data are stored. Obscure part of the data
- 44. Pseudonymity: Tokenization 👍 Popular libraries like Faker. 👍 All original data is replaced. 👎 No way to recover the original data. Replace real data with fake data
- 45. Pseudonymity: Blurring 👍 95% of this image is left unblurred. 👎 Possible to reverse blurring. Blur a subset of the data
- 46. Pseudonymity: Encryption 👍 The original data can be recovered. 👍 Manage fewer permissions downstream. 👎 Asymmetric vs Symmetric trade-offs. Two-way transformation of the data
- 47. AWS Key Management Service ● Generate a new data key for encrypting and decrypting data protected by a master key. ● Or manually rotate the master key and re-encrypt the data. Automate key creation and rotation
- 48. Encrypted Columns ● pgcrypto allows us to encrypt sensitive columns before the data lands in our S3 data lake. ● This example is encrypting the birth_date column in Postgres. Postgres pgcrypto
- 49. “Last Mile” Decryption ● Access to encrypted columns is limited to analysts with the encryption key. ● This example is decrypting the birth_date column in Redshift. Decrypt sensitive data at query time
- 50. Encrypted Column Problems Some things to consider... 1. Symmetric or Asymmetric encryption scheme? 2. Should we manually rotate our master key? 3. How many keys should we use and how should they be organized? 4. Should our analysts and data scientists need to think about keys? 5. When and how do we re-encrypt data? When an employee with access to keys leaves the company?
- 51. 3 Big Ideas 1. Privacy has a strange history. 2. Privacy-first systems are designed by people with a professional ethic. 3. Privacy can be automated away. Automating security in your data pipeline privacy