![[Velocity Conf 2017 NY] How Twitter built a framework to improve infrastructure utilization and efficiency at scale](https://image.slidesharecdn.com/velocityconf-171013225240/85/Velocity-Conf-2017-NY-How-Twitter-built-a-framework-to-improve-infrastructure-utilization-and-efficiency-at-scale-8-320.jpg)
![[Velocity Conf 2017 NY] How Twitter built a framework to improve infrastructure utilization and efficiency at scale](https://image.slidesharecdn.com/velocityconf-171013225240/85/Velocity-Conf-2017-NY-How-Twitter-built-a-framework-to-improve-infrastructure-utilization-and-efficiency-at-scale-58-320.jpg)
![[Velocity Conf 2017 NY] How Twitter built a framework to improve infrastructure utilization and efficiency at scale](https://image.slidesharecdn.com/velocityconf-171013225240/85/Velocity-Conf-2017-NY-How-Twitter-built-a-framework-to-improve-infrastructure-utilization-and-efficiency-at-scale-74-320.jpg)
[Velocity Conf 2017 NY] How Twitter built a framework to improve infrastructure utilization and efficiency at scale
- 1. How built a framework to improve infrastructure resource utilization at scale
- 2. ★ Sr. Systems Engineer @Twitter ★ Proud Member of @TwitterWomen, @WomenWhoCode Iam@VinuCharanya Hello!
- 3. 3 1 2 3 4 History & Context Chargeback @Twitter Kite - Service Lifecycle Manager Impact & Future Work Agenda
- 9. INFRASTRUCTURE & DATACENTER MANAGEMENT CORE APPLICATION SERVICES TWEETS USERS SOCIAL GRAPH PLATFORM SERVICES SEARCH MESSAGING & QUEUES CACHE MONITORING AND ALERTING INGRESS & PROXY FRAMEWORK/ LIBRARIES FINAGLE (RPC) SCALDING (Map Reduce in Scala) HERON (Streaming Compute) JVM MANAGEMENT TOOLS SELF SERVE SERVICE DIRECTORY CHARGEBACK CONFIG MGMT DATA & ANALYTICS PLATFORM INTERACTIVE QUERY DATA DISCOVERY WORKFLOW MANAGEMENT INFRASTRUCTURE SERVICES MANHATTAN BLOBSTORE GRAPHSTORE TIMESERIESDB S T O R A G E MESOS/AURORA HADOOP C O M P U T E MYSQL VERTICA POSTGRES D B / D W DEPLOY (Workflows)
- 11. Number of Servers MESOS/AURORA HADOOP MANHATTAN 67% How to get visibility into resources used by individual jobs & datasets?
- 12. Number of Servers MESOS/AURORA HADOOP MANHATTAN 67% How to attribute resource consumption to teams/organization?
- 13. Number of Servers MESOS/AURORA HADOOP MANHATTAN 67% How do you incentivize the right behavior to improve efficiency of resource usage?
- 15. Chargeback @Twitter Ability to meter allocation & utilization of resources
- 16. Chargeback @Twitter Ability to meter allocation & utilization of resources per service, per project, per engineering team
- 17. Chargeback @Twitter Ability to meter allocation & utilization of resources per service, per project, per engineering team to improve visibility & enable accountability
- 18. Features Supports diverse Infra Services Chargeback @Twitter 18 Meters abstract resources at daily granularity Detailed Reports
- 19. 19 Chargeback @Twitter 1. Resource Catalog: Consistent way to inventory infrastructure resources Support diverse Infrastructure and Platform Services
- 20. 20 Chargeback @Twitter 1. Resource Catalog: Consistent way to inventory infrastructure resources • Resource Fluidity: Support primitive (CPU) and abstract resource (“Tweets / second”). Extend existing resource Support diverse Infrastructure and Platform Services
- 21. 21 Chargeback @Twitter 1. Resource Catalog: Consistent way to inventory infrastructure resources • Resource Fluidity: Support primitive (CPU) and abstract resource (“Tweets / second”). Extend existing resource 2. Resource <> Client Identifier Ownership: Map of client identifier to an owner to enable accountability Support diverse Infrastructure and Platform Services
- 22. OFFER MEASURE COST RESOURCE CATALOG ENTITY MODEL
- 23. OFFER MEASURES OFFER MEASURE COST 1:N RESOURCE CATALOG ENTITY MODEL
- 24. PROVIDER INFRASTRUCTURE SERVICE OFFERINGS OFFER MEASURES OFFER MEASURE COST 1:N 1:N 1:N 1:N RESOURCE CATALOG ENTITY MODEL
- 25. TWITTER DC/ PUBLIC CLOUD COMPUTE CORE-DAYS $X PROVIDER INFRASTRUCTURE SERVICE OFFERINGS OFFER MEASURES OFFER MEASURE COST 1:N 1:N 1:N 1:N RESOURCE CATALOG ENTITY MODEL
- 26. TWITTER DC/ PUBLIC CLOUD COMPUTE CORE-DAYS $X PROVIDER INFRASTRUCTURE SERVICE OFFERINGS OFFER MEASURES OFFER MEASURE COST 1:N 1:N 1:N 1:N TWITTER DC STORAGE GB- RAM PROCESSING CLUSTER FILE ACCESSES … … GB- RAM FILE ACCESSE S … … $X $Y …$M $N… … RESOURCE CATALOG ENTITY MODEL
- 27. { measures: [ { "measure_id": 1, "measure_label": "core-days", "measure_unit_label": "per 1 core-day", "offering_id": 1, "offering_label": "Compute", "infrastructure_id": 1, "infrastructure_name": "Aurora" }, { "measure_id": 2, "measure_label": "machine-days", "measure_unit_label": "per 1 machine-day", "offering_id": 2, "offering_label": "zone:aquila", "infrastructure_id": 8, "infrastructure_name": "Physical Infrastructure", }, { /api/1/measures Chargeback @Twitter
- 28. So, how do you incentivize the right behavior to improve efficiency of resource usage?
- 29. Pricing is one way…
- 30. Operational Overhead Headroom Production Used Cores Non-Prod Used Cores Cost of Physical Server ($X / day) Total available Cores Quota Buffer (Underutilized Quota) Container Size Buffer (Underutilized Reservation) Total Cost of Ownership for Aurora $X core-day
- 31. Operational Overhead Headroom Production Used Cores Non-Prod Used Cores Cost of Physical Server ($X / day) Total available Cores Quota Buffer (Underutilized Quota) Container Size Buffer (Underutilized Reservation) Total used Cores Total Cost of Ownership for Aurora $X core-day
- 32. Operational Overhead Headroom Production Used Cores Non-Prod Used Cores Cost of Physical Server ($X / day) Total available Cores Quota Buffer (Underutilized Quota) Container Size Buffer (Underutilized Reservation) Total used Cores Excess Cores (incl. DR, Spikes, Overallocation)Total Cost of Ownership for Aurora $X core-day
- 33. Operational Overhead Headroom Production Used Cores Non-Prod Used Cores Cost of Physical Server ($X / day) Total available Cores Quota Buffer (Underutilized Quota) Container Size Buffer (Underutilized Reservation) Total used Cores Excess Cores (incl. DR, Spikes, Overallocation) Cores used by platform for operations & maintenance Total Cost of Ownership for Aurora $X core-day
- 34. Operational Overhead Headroom Production Used Cores Non-Prod Used Cores Cost of Physical Server ($X / day) Total available Cores Quota Buffer (Underutilized Quota) Container Size Buffer (Underutilized Reservation) Total used Cores Excess Cores (incl. DR, Spikes, Overallocation) Cores used by platform for operations & maintenance Total Cost of Ownership for Aurora $X core-day
- 35. Our team would be …
- 36. Features Supports diverse Infra/Platform Services Chargeback @Twitter 36 Meters abstract resources at daily granularity Detailed Reports
- 37. 37 Chargeback @Twitter INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG REPORT REPORT Metering Pipeline (ETL Job) IDENTIFIER OWNERSHIP MAPPING Metrics Ingestor DATA FIDELITY Metering Pipeline (ETL Job)
- 38. 38 Chargeback @Twitter INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG REPORT REPORT Metering Pipeline (ETL Job) IDENTIFIER OWNERSHIP MAPPING Schema(client_identifier, offering_measure, volume, metadata, timestamp) DATA FIDELITY Metering Pipeline (ETL Job)
- 39. 39 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG IDENTIFIER OWNERSHIP MAPPING REPORT REPORT Transformer DATA FIDELITY Metering Pipeline (ETL Job)
- 40. 40 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG IDENTIFIER OWNERSHIP MAPPING REPORT REPORT 1. Resolve Ownership DATA FIDELITY Metering Pipeline (ETL Job)
- 41. 41 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG IDENTIFIER OWNERSHIP MAPPING REPORT REPORT 2. Cost Computation DATA FIDELITY Metering Pipeline (ETL Job)
- 42. 42 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG DATA FIDELITY REPORT REPORT IDENTIFIER OWNERSHIP MAPPING Data Fidelity & Reporting Metering Pipeline (ETL Job)
- 43. 43 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG REPORT REPORT IDENTIFIER OWNERSHIP MAPPING 1. Verify Data Integrity & Fidelity DATA FIDELITY Metering Pipeline (ETL Job)
- 44. 44 Chargeback @Twitter Metering Pipeline (ETL Job) INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 INGEST METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG REPORT REPORT IDENTIFIER OWNERSHIP MAPPING 2. Alert when things don’t seem the way it should be DATA FIDELITY Metering Pipeline (ETL Job)
- 45. 45 Chargeback @Twitter INFRASTRUCTURE SERVICE 1 INFRASTRUCTURE SERVICE 2 EXPORT METRICS RAW FACT TRANSFORMER RESOLVED FACT RESOURCE CATALOG IDENTIFIER OWNERSHIP DATA FIDELITY REPORT REPORT Metering Pipeline (ETL Job)
- 46. Features Supports diverse Infra/Platform Services Chargeback @Twitter 46 Meters abstract resources at daily granularity Detailed Reports
- 47. 47 Chargeback @Twitter Customers Infrastructure & Platform Operators Overall Cluster Growth Allocation v/s Utilization of resources by Client/Tenant Finance & Execs Budget v/s Spend per Org Infrastructure PnL Overall Efficiency & Trends Service Owners & Developers Team Bill Per Service Allocation vs. Utilization of Resources Reports Customers Infrastructure & Platform Operators Overall Cluster Growth Allocation v/s Utilization of resources by Client/Tenant Finance & Execs Budget v/s Spend per Org Infrastructure PnL Overall Efficiency & Trends
- 49. 49 Chargeback @Twitter Customers Infrastructure & Platform Operators Overall Cluster Growth Allocation v/s Utilization of resources by Client/Tenant Finance & Execs Budget v/s Spend per Org Infrastructure PnL Overall Efficiency & Trends Service Owners & Developers Team Bill Per Service Allocation vs. Utilization of Resources Reports
- 50. CHARGEBACK BILL FOR A TEAM
- 51. CHARGEBACK DRILLDOWN FOR A TEAM
- 52. Features Supports diverse Infra/Platform Services Chargeback @Twitter 52 Meters abstract resources at daily granularity Detailed Reports
- 53. 53 1 2 3 4 Learnings Chargeback @Twitter Invest in data Fidelity Accurate Ownership Mapping Logical grouping of resources Change History • Trust in data is most important. • Invest in monitoring & alerting for data inconsistencies • Leverage this for detecting abnormal increase/decrease and notify users • Static mappings go out of date quickly • Invest in systems (ex, Kite) for users to manage it themselves • Identifiers were too granular and teams were too broad. • Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain • Unit prices change over time • Orgs / Teams change over time • Resources get added / removed • Change history is essential for consistency which is used for CAP planning
- 54. 54 1 2 3 4 Learnings Chargeback @Twitter Invest in data Fidelity Accurate Ownership Mapping Logical grouping of resources Change History • Trust in data is most important. • Invest in monitoring & alerting for data inconsistencies • Leverage this for detecting abnormal increase/decrease and notify users • Static mappings go out of date quickly • Invest in systems (ex, Kite) for users to manage it themselves • Identifiers were too granular and teams were too broad. • Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain • Unit prices change over time • Orgs / Teams change over time • Resources get added / removed • Change history is essential for consistency which is used for CAP planning
- 55. 55 1 2 3 4 Learnings Chargeback @Twitter Invest in data Fidelity Accurate Ownership Mapping Logical grouping of resources Change History • Trust in data is most important. • Invest in monitoring & alerting for data inconsistencies • Leverage this for detecting abnormal increase/decrease and notify users • Static mappings go out of date quickly • Invest in systems (ex, Kite) for users to manage it themselves • Identifiers were too granular and teams were too broad. • Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain • Unit prices change over time • Orgs / Teams change over time • Resources get added / removed • Change history is essential for consistency which is used for CAP planning
- 56. 56 1 2 3 4 Learnings Chargeback @Twitter Invest in data Fidelity Accurate Ownership Mapping Logical grouping of resources Change History • Trust in data is most important. • Invest in monitoring & alerting for data inconsistencies • Leverage this for detecting abnormal increase/decrease and notify users • Static mappings go out of date quickly • Invest in systems (ex, Kite) for users to manage it themselves • Identifiers were too granular and teams were too broad. • Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain • Unit prices change over time • Orgs / Teams change over time • Resources get added / removed • Change history is essential for consistency which is used for CAP planning
- 57. 57 1 2 3 4 Learnings Chargeback @Twitter Invest in data Fidelity Accurate Ownership Mapping Logical grouping of resources Change History • Trust in data is most important. • Invest in monitoring & alerting for data inconsistencies • Leverage this for detecting abnormal increase/decrease and notify users • Static mappings go out of date quickly • Invest in systems (ex, Kite) for users to manage it themselves • Identifiers were too granular and teams were too broad. • Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain • Unit prices change over time • Orgs / Teams change over time • Resources get added / removed • Change history is essential for consistency which is used for CAP planning
- 59. SERVICE IDENTITY MANAGER RESOURCE PROVISIONING MANAGER DASHBOARD (SINGLE PANE OF GLASS) REPORTING INFRASTRUCTURE SERVICEINFRASTRUCTURE SERVICEINFRASTRUCTURE SERVICEINFRASTRUCTURE & PLATFORM SERVICE SERVICE LIFECYCLE WORKFLOWS METADATA RESOURCE QUOTA MANAGEMENT METERING & CHARGEBACK CLIENT IDENTITY PROVIDER APIS & ADAPTERS
- 60. 10,000+ClientIdentifiers 1,000+ Projects 100+ Teams 8 InfrastructureServices 60 Kite @Twitter
- 61. 61 Kite @Twitter Identity System: Built a consistent way to group client identifiers of different infrastructure services into a project and enabled ownership • Capture Org Structure: Support org structure changes, project transfer workflows to ensure up-to-date ownership of identifiers • Unify client identifier provisioning workflow: Enables single source of truth and reduces operator pain around provisioning and managing client identifiers. Client Identifier Management
- 62. IDENTITY ENTITY MODEL <INFRA, CLIENTID> <Aurora, tweetypie.prod.tweetypie> <Aurora, ads- prediction.prod.campaign-x>
- 63. IDENTITY ENTITY MODEL SERVICE/ SYSTEM ACCOUNT <INFRA, CLIENTID> 1:N tweetypie <Aurora, tweetypie.prod.tweetypie> ads-prediction <Aurora, ads- prediction.prod.campaign-x>
- 64. BUSINESS OWNER TEAM PROJECT SERVICE/ SYSTEM ACCOUNT <INFRA, CLIENTID> 1:N 1:N 1:N 1:N INFRASTRUCTURE TWEETYPIE tweetypie tweetypie <Aurora, tweetypie.prod.tweetypie> ADS PREDICTION prediction ads-prediction <Aurora, ads- prediction.prod.campaign-x> REVENUE IDENTITY ENTITY MODEL
- 65. BUSINESS OWNER TEAM PROJECT SERVICE/ SYSTEM ACCOUNT <INFRA, CLIENTID> 1:N 1:N 1:N 1:N INFRASTRUCTURE TWEETYPIE tweetypie tweetypie <Aurora, tweetypie.prod.tweetypie> ADS PREDICTION prediction ads-prediction <Aurora, ads- prediction.prod.campaign-x> REVENUE IDENTITY ENTITY MODEL Entities are time varying dimensions
- 66. Impact
- 68. CLAIM OWNERSHIP
- 70. PROJECT METADATA
- 72. Future Work
- 73. 73 Future Work Impact & Future Work 1 2 Capacity Planning Extend Quota Manager • Provide historic trends and help with forecast of capacity • Onboard Hadoop, Storage and other systems 3 Enable project deprecation • Detect unused resources, notify users, trigger deprecation process based on policy
- 75. 75 1 2 Future Work Impact & Future Work Capacity Planning Extend Quota Manager • Provide historic trends and help with forecast of capacity • Onboard Hadoop, Storage and other systems 3 Enable project deprecation • Detect unused resources, notify users, trigger deprecation process based on policy
- 76. 76 1 2 Future Work Impact & Future Work Capacity Planning Extend Quota Manager • Provide historic trends and help with forecast of capacity • Onboard Hadoop, Storage and other systems 3 Enable project deprecation • Detect unused resources, notify users, trigger deprecation process based on policy
- 77. @VinuCharanya