DB Engineering - From Antiquated to Engineer
1 like1,091 views
The document discusses the evolution of database administration from yesterday's gatekeeper role to today's database engineering role. A database engineer provides expertise to development teams to help them meet performance metrics and troubleshoot issues. The document outlines how database engineering aligns with devops goals and processes. It proposes that database engineers follow a craft approach focused on protecting data, eliminating waste, and data-driven decision making.
DB Engineering - From Antiquated to Engineer
- 1. from antiquated to engineer AllYourBase, November 2015 Laine Campbell, DB Architect/Engineer lainevc@gmail.com @lainevcampbell September, 2015
- 5. database engineering 5 shared services for: cross-functional development teams operations engineering systems engineering
- 6. database engineering 6 provides expertise to other team’s processes to facilitate all work (for example, SRE teams) understands and teaches data access and storage nuances to ensure all performance metrics can be met anchors teams with expertise for troubleshooting, recovery and other tasks requiring depth, not breadth
- 8. polyglot persistence 8 relational is not the end of the line data exists as part of a pipeline function dictates form we cannot predict all uses
- 10. virtualization and cloud 10 forces horizontal scaling forces designing for resilience elasticity drives new data storage management by API enables continuous deployment
- 11. infrastructure as code 11 forces standardization enables immutability forces us to learn to code we start building platforms changes become deployments
- 12. continuous delivery © 2014 Pythian 12 brings us to the source code control paradigm we must be teachers, not gatekeepers testing and compliance become top priorities
- 13. devops cultures 13 lean manufacturing defines our workflows theory of constraints (we are a constraint) tighter feedback loops require organizational shifts experimentation and controlled failure shift architecture and process design integration drives empathy
- 14. database engineers: O.G. devops 14 DEV OPS DBE shared goals, tools and processes
- 15. 15
- 16. db engineer’s manifesto 16 it’s all about the mission protect the data eliminate waste data-driven decision making databases are not special eliminate the barriers between software and ops
- 17. the craft 17
- 18. maslow’s hierarchy of needs 18 physiological safety love and belonging esteem self-actualization with loving credit and glory to Charity F. Majors
- 19. physiological 19 physiological does it run and service your application? is it stable? can data be backed up and restored? can basic levels of service be monitored?
- 20. safety 20 physiological safety do I have n+1 available copies? is failover built, documented and tested? is my data distributed to multiple physical locations? can I detect, and recover from corruption? is access managed via roles, separation of duties and appropriate authorization? is appropriate data encrypted in transport and at rest?
- 21. love and belonging 21 physiological are database changes part of integration and deployment processes? is continuous testing done with proper data sets? is infrastructure managed via configuration management, automation and orchestration? is all database interaction done via scripted, repeatable processes? is consistency of configuration monitored and managed? safety love and belonging
- 22. esteem 22 physiological safety love and belonging esteem are all relevant metrics being captured at appropriate granularity? can problems be detected preemptively? are capacity correlations understand and monitored? are developers trained and knowledgeable on the datastore? can the system automatically remediate frequent and simple faults?
- 23. self-actualization 23 physiological safety love and belonging esteem self-actualization are you using the right datastore for the right job? are your developers in tight feedback loops with the datastore? is experimentation and data driven decision making supported by the datastore? is the mission fully supported?
- 24. a day in the life... 24 selecting datastores to add to production platform catalogs dbms and feature education to software, systems and operations engineers integrating with company-wide services (reporting, monitoring, config mgmt etc…)
- 25. a day in the life... 25 validating acceptable configurations testing and benchmarking new versions, features and configurations documenting and sharing standards
- 26. deploying infrastructure 26 configuration management definition orchestration enablement change and deployment test automation building self-service tools for the team
- 27. deploying software 27 agile participation with engineers using version control for schemas and metadata more flexible data model approaches teach your engineering teams how to assess risk, performance and impact support test planning, building and execution
- 28. a day in the life... 28 attending scrums, grooming and planning committing deployment scripts to the codebase building, running and automating test cases monitoring commits for key areas that need your attention
- 29. a day in the life… (cont) 29 providing new or modified recipes for CM and automation pairing with, and teaching engineers, iterating on the schema
- 30. the data must flow 30 data brokers pub/sub systems real-time analytics batch processing cache population, use, and flushing search stores
- 31. backup and recovery 31 this has not changed, we live and die by the safety of our data borrow ideas of continuous deployment, for continuous recovery testing build backup and recovery into every possible process
- 32. a day in the life... reviewing current workloads and tuning managing escalations on DB issues continuous improvement writing, testing and performing change plans as part of the deployment process 32
- 33. 33 how?
- 34. software engineering 34 bring DBEs into your agile workflows learn the software versioning system learn the testing frameworks study the language, the framework, the drivers and the ORMs teach, teach teach!
- 35. systems engineering 35 design your data pipelines together collaborate on data standards together learn the entire stack to moderate proficiency understand the entire flow teach, teach, teach!
- 36. operations engineering 36 build your scripting chops learn the build and deploy frameworks learn the configuration management, automation and orchestration layers and patterns learn operational visibility layers and patterns plan, execute and learn from failures
- 37. further deep dives 37 understand the statistics and math around distributions, anomaly detection and correlation write and push code! answer the customer service phones dive into your network layers teach everyone about the data