Байки із пожежного депо або як працює Big Data в Sigma Software, Денис Пишьєв,
0 likes84 views
This document provides advice for teams working on big data projects. It discusses the importance of knowing and trusting your team members, not over-relying on any single person, being aware of risks posed by the technical platform, learning your product inside and out, and securely handling credentials and access. Specific recommendations include listening to engineers, allowing flexibility in work, having backup plans, experimenting with default settings, monitoring data flows, and using information security best practices like separating code and credentials.
![Learn your product: case of [object Object]](https://image.slidesharecdn.com/bigdatadepotailsdenyspushiev-190531123111/85/Big-Data-Sigma-Software-11-320.jpg)
Байки із пожежного депо або як працює Big Data в Sigma Software, Денис Пишьєв,
- 1. Tales from fire depot: how to do BigData and survive
- 2. About me: ● Denis Pyshev, Twitter: @gemelen ● Senior Software Engineer, Sigma Software ● Software development in general, FinTech, AdTech, 8+ years ● Data Platform / Datawarehouse team, Verizon Media Group (Oath/AoL+Yahoo), 3+ years ● Lviv Scala User Group
- 3. Team challenges ● Know (and trust) your team ● Do not throw bus factor under the bus ● Beware of your platform ● Learn your product ● Secure your stuff
- 4. Know (and trust) your team ● Your team members are not code monkeys (I hope so) ● They could know better ● At least try to listen to them ● Let your engineers make things they would like (it will save money too)
- 5. Know (and trust) your team: case of GDPR
- 6. Do not throw bus factor under the bus ● People go to vacation from time to time (do they?) ● Shit happens, no exceptions ● Shit would probably happen when person in charge is out of town
- 7. Do not throw bus factor under the bus: laptop stays home
- 8. Beware of your platform ● Collect platform knowledge ● Experiment and share results (at least within the team) ● Check if default values are viable in your setup
- 9. Beware of your platform: case of compression
- 10. Learn your product ● Domain knowledge does worth to get ● Collected data is worthless without its consumption ● Your product doesn’t fly in free space: monitor ins and outs ● Create your own monitoring tools on top of monitoring services ● We did so: check github.com/verizonAdPlatforms/bandar-log
- 11. Learn your product: case of [object Object]
- 12. Secure your stuff ● Never ever store any credentials in repositories, no matter how private they are – someday they would leak ● “The response to the credentials leakage was absolutely overwhelming for us. Within less than one-minute foreign parties tried to take advantage of the leaked credentials. Within the 6 minutes in which the credentials were valid we received connections from different Anonymizing Networks and from China.” - ict.swisscom.ch ● Use infosec/devops techniques to separate code, configuration and credentials ● Use 2FA
- 14. Q&A