Hitchhikers guide to_data_center_facility_ops
- 2. The Hitchhikers Guide to Data Center Facility Operations The following is a very simplistic list of the types of questions/criteria any organization should consider in evaluating the outsourcing of its critical facility operation functions. This list (in no particular order) should be helpful to anyone looking to compile an RFP for these services, or serve as a guideline for evaluating and improving your current facility operations. Tip for reading this guide: This document is written from the prospective client‟s perspective (as if it were an actual RFP). „You‟ refers to the vendor unless otherwise noted. Standard Considerations for any Request for Proposal Service and Delivery Overview .................................................................................................................... 2 Planning / Preparation ............................................................................................................................ 2 Execution / Implementation (Service delivery) ...................................................................................... 2 Measurement (measuring the success of implementation and beyond)............................................. 3 Methodology Overview ................................................................................................................................. 3 Personnel Management .......................................................................................................................... 4 Training ..................................................................................................................................................... 5 Documentation......................................................................................................................................... 6 Processes & Procedures.......................................................................................................................... 6 Quality Systems ........................................................................................................................................ 8 Support Systems (CMMS/EDMS)............................................................................................................ 9 Value-added Innovations .......................................................................................................................... 10 Cost Management / Cost Savings ....................................................................................................... 10 Efficiency Improvements (Energy Efficiency) ....................................................................................... 10 Quality & Process Improvements ......................................................................................................... 10 Customer Service / Satisfaction .............................................................................................................. 11 Organizational Values & Policies ............................................................................................................. 12
- 3. Service and Delivery Overview Planning / Preparation Describe in detail the process by which you (the vendor) will prepare and plan for implementation of services, including all activities leading up to execution and delivery of those services. Bidding Approach: What is your overall approach in bidding on these services? Project Kickoff: How will you initiate/kick off the project, including identification of key team members for the vendor and client, communications protocol and key objectives, such as defining service deliverables, developing a tentative transition schedule and identifying key metrics? Project Management: Define your approach to critical facility project management and how you would manage facility projects (both in scope and out of scope). Include your approach to project related controls. Operational Program Development: How will you develop the site(s)‟s operational program, including an O&M program, staffing plan, and implementation plan? Implementation Plan: Describe in detail your comprehensive roll-out plan for our critical facility operations. Define the interaction and duties between the client and vendor teams. Provide details on all activities and a comprehensive timeline of services. Define your resource requirements. Identify risks associated with the conversion of services and address how they will be mitigated and who is accountable for the execution of the process (client or vendor). Execution / Implementation (Service delivery) Provide a detailed description of how you will execute the implementation plan and deliver services. Be detailed in the overall procedures and methods by which you will deliver these services. Provide details of the type of subcontractors you propose to use and how these services will be managed. Displacement of Personnel: If existing personnel are to be displaced, how will you manage the process? Subcontractors: What, if any services do you intend to subcontract as part of this response and why? Subcontractor Management: Describe how you manage subcontracting entities, including how you evaluate subcontractor performance. Changes in Scope: Should there be a need to change the pre-existing scope of work (such as facility expansions, consolidations, equipment additions or deletions, additions in number of sites, etc.), how do you approach these changes? What methods are used in managing these scope changes? Can you describe examples of how this was handled with other 2
- 4. clients? What are the associated factors that normally change the proposed scope of services over a given contract term? Team Make-up: Please define your project team members. Include their roles in the program, titles, level of experience/certifications, individual expertise and overall education levels. Describe the back office support and subject matter experts that are available to support onsite operations staff (outside those located at the facility). Measurement (measuring the success of implementation and beyond) Discuss how the quality and success of service delivery will be evaluated initially and adjusted throughout the proposed duration of the contract. Post implementation Review: After the implementation phase is complete, how will it be evaluated initially and continually to ensure success and client satisfaction? Key Performance Indicators (KPI): Will KPI metrics be established for the implementation review? Will KPI metrics be used for the duration of the contract? If so, on what timeline/frequency? What typical KPIs do you use? Reports: What standard reports do you provide? How frequently are these reports provided to the client? Please provide examples of all of the general reports used most frequently. Will there be an additional fee for any/all Ad Hoc reporting? Methodology Overview Describe in detail your operational methodology. That is, how do you effectively wrap all of the components of a successful critical facility operation into a succinct process that becomes the company‟s doctrine? *Tip!* Look beyond the elements of the program itself and dive into how those elements connect and interact with each other. Things to Consider: Beware of jargon. Ex. Terminology like MOP (method of procedure) and SOP (standard operating procedure) is used frequently by most vendors. By nature, the idea behind standardization and procedures is to ensure uniformity, accuracy and to avoid mistakes. But in a data center environment, these MOPs and SOPs are living documents that should evolve with time, experience and education. How will the vendor ensure that a “checkmark syndrome” scenario is avoided? (Ex. A procedure or process becomes perfunctory and is completed simply to check it off the list. Rather a procedure or process should be followed and actively evaluated for effectiveness in meeting its intended goal.) Look for: how the vendor ensures continuous improvement. If improvements are made, how is the quality ensured? How do changes and updates trickle down to training and 3
- 5. documentation? How is this engrained into the culture of the organization and in its personnel? Personnel Management What is your overall approach to personnel management, including evaluating team size, skill set requirements, recruiting, technical qualification, background screening, hiring and placement, retention and career progression? Staff Determination: How do you assess and determine staff levels in number as well as technical capability? Staff Capabilities: How do you validate and ensure proper technical capabilities of your staff members for our organization‟s facilities? Assessment of Current Staff: How do you assess these individuals for their level of skill and position within your program? Turnover Rates: What is your average annual turnover rate (by percentage) for members supporting critical facility environs? Can you provide statistics over the past three years? Plan for Potential Displacement of Staff: With the potential for loss of staff through natural attrition or general displacement, how do you manage this and make this successful for the client? Staff Replacement: What program do you have in place when a staff member needs to be replaced? Does this plan provide for continuity of service during this transition? If so, how? What processes and methods are in place to ensure there is ample transfer of site knowledge by incoming/outgoing staff members? Career progression: What employee career and skill development programs do you have as part of your program? Things to Consider: Recruiting talented individuals in specific technical disciplines that are capable of working together as a team is an extremely challenging task that requires experience and extensive knowledge of critical facility operations. Prospective team members need to be carefully screened not only with traditional background checking, but also to qualify their technical, administrative and communications capabilities, all of which are crucial skills in Critical Facility Operations. Simply identifying qualified personnel is only the first step. They also need to be smoothly transitioned into the critical environment and provided with the support and career opportunities that will ensure that their talent and experience is retained and developed. 4
- 6. Training Describe your overall approach to a training program, including a discussion on general training, critical systems training, evaluation, certifications and site specific training? Critical Systems: What type of critical systems training do you provide? Describe how safety, site operations, emergency response, maintenance procedures, and third-party vendor management will be addressed. Program structure: Is this program structured or unstructured? In-house or outsourced? Supplemented online, in person, hands-on? Drills and scenario training: How does the facility operations team hone its skill in a live data center environment? Do they drill and perform scenario training? If so, how and how often? Certification: Describe evaluation and examination of skills. What type of certifications does your proposed vendor team hold? Assessment: How often will the proposed vendor team re-certify or get assessed for improved/continued skill levels? Site-specific training: What site-specific training do you suggest for our organizations facility(s)? Describe your overall approach to site specific training scenarios? Things to Consider: Commonly found training program elements are turnover rate, vendor training and on the job training. However, in a critical environment this simply isn‟t enough. What are missing are detailed written procedures, thorough training and certification, quality assurance and continuous process improvement. In our experience, an effective training program is multi- level, with each level corresponding to a specific operational action or activity. Personnel knowledge must be thoroughly evaluated and denoted by certification level. Beyond simple training and testing, simulated scenario drills should be conducted frequently to ensure retention of knowledge and ability to execute. Periodically, processes, procedures and personnel should be reviewed and re-qualified. Most importantly, there should be a feedback loop to incorporate lessons learned and trainee input. 5
- 7. Documentation Describe your overall approach to documentation, including types, organization and management. Discuss areas, such as a) as-built and record drawings b) asset database c) preventative maintenance scope of work d) maintenance schedule e) critical facility work rules f) safety program g) facility reports h) walkthrough checklist Things to Consider: Vendor turnover documentation can be an impressive volume of material, but while it is a vital component of the operation, it hardly constitutes the totality of what‟s needed to effectively sustain operations. What‟s typically missing are the detailed procedures and reports that the critical environments team will need to perform tasks, such as facility walkthroughs, routine operations, preventative maintenance, corrective maintenance, and emergency response. As-built documentation, even where it exists and is accurate, is a static picture of the facility at a single point in time. Accurate, up-to-date record drawings are vital to safe and reliable facility operations. Seemingly simple or obvious information such as equipment lists, scopes of work for equipment maintenance and maintenance schedules are frequently missing, inaccurate or inadequate. Since this is foundational information needed for a comprehensive maintenance program, incorrectly assuming that it has been properly collected and organized, either by vendors or in-house personnel is incredibly risky. Processes & Procedures Describe your formalized approach to policies & procedures in detail. Provide sample documentation where relevant. Change Controls: What is your approach to events such as moves/add/changes and change controls in general? Maintenance Programs: How do you develop a thorough/comprehensive maintenance program for facilities? Procedure Training: Are processes and procedures incorporated into training? If so, how? MOP/SOP/EOP Examples: Please provide documentation examples of Methods of Procedure (MOPs), Standard Operating Procedures (SOPs) and Emergency Operating Procedures (EOPs). 6
- 8. Policy & Procedure Examples: Provide/demonstrate samples of existing policy and procedure manuals. Things to Consider: Change Control is used in critical environments to ensure that all system changes are assessed and approved prior to their implementation, and that the result of the change conforms to the predicted and required result. This can only be accomplished with a formal set of procedures and processes that follow generally accepted guidelines such as ITIL change and configuration management. Virtually everything that takes place in the data center should have a written procedure. Procedures can be utilized in a variety of ways and have specialized formats that are specific to the particular task at hand. The most commonly used procedures are: Standard Operating Procedure (SOP) A SOP can be functional or administrative. It details a fixed operating procedure and can be referenced whenever needed. Method of Procedure (MOP) A MOP is the detailed, step-by-step procedure that is used when working on or around any piece of equipment that has the ability to directly or indirectly impact the critical load. A library of MOP‟s should exist for scheduled maintenance operations, and should be written for corrective maintenance and installation activities as well. Emergency Operating Procedure (EOP) An EOP is an emergency response procedure for a potential or previously experienced failure mode. It covers how to get to a safe condition, restore redundancy and isolate the trouble. Vendor Management When vendors are engaged in work on or around the critical systems, unnecessary risk is introduced unless a comprehensive program is in place that begins with vendor selection and includes work specification, procedural controls, work supervision and service documentation. Emergency Response Emergency response and reaction protocols are essential to the minimization of system downtime. Unpredictable events will occur no matter how careful the preparation. A well designed and up-to-date escalation process can prevent or mitigate damage, while detailed incident reporting, failure analysis and a lessons-learned program will help prevent future occurrences. 7
- 9. Quality Systems What quality systems do you employ to help meet the 100% availability (zero defects) requirement for mission critical facilities? Describe your overall approach to ensuring the quality, such as quality assurance (QA), quality control (QC) and quality improvement (QI). Quality Program: What is your QA/QC/QI plan in regards to the review, creation, and constant adjustment to operations procedures (SOPs, MOPs, EOPs, etc)? Quality Assurance: Describe the process by which you will create and modify processes and procedures, including procedure scope, process and procedure training and document control? Quality Control: Describe the process by which you will control quality, such as reviewing and approving procedures, reviewing incidents, and performing operational quality reviews, checks and inspections. Quality Improvement Techniques: Does the vendor follow business management strategies such as LEAN, Six Sigma or TQM? Improvement Examples: Please demonstrate examples where your program has improved your client‟s critical operations. Be as descriptive as possible. Things to Consider: Quality Assurance processes are employed to ensure that errors are not introduced into the system. Quality Control measures are taken at various stages of the process in an attempt to identify problems that could potentially lead to system failure before they happen. The level of excellence required to achieve zero downtime is not easily attainable. No process or procedure is perfect, particularly in its early stages. It‟s crucial to have a plan for continuous process improvement in place to provide a mechanism for fine tuning the program. For instance, all MOP‟s should have a feedback section that is used to document any variance that was noted in how the procedure was performed vs. how it was written. Suggestions for improvements can be rolled immediately into a new version of the procedure and run through QA. Important lessons can be documented in a Lessons Learned Report and distributed throughout the organization. 8
- 10. Support Systems (CMMS/EDMS) Provide an overview of the vendor‟s approach to operational support systems, such as electronic document management systems (EDMS) and computerized maintenance management systems (CMMS). Management Systems Integration: Assuming the client has a current maintenance management system, how would you work with or integrate your preferred systems with the client‟s existing system? Will you utilize our pre-existing maintenance management system? What are your recommendations as it relates to current CMMS products that are available? Is the proposed system web or server based? Asset Data Management System: What asset data management system or CMMS software do you prefer to use for asset management and reporting purposes? Are you willing / able to work with our existing or preferred system? Maintenance Management Systems Support: What technical staff/support teams are available and proposed in this response as it relates to your CMMS products? How would the support change or differ if the system is web vs. server based? CMMS System upgrades: Within the proposed/selected computerized maintenance management product, how often do you upgrade this system? Things to Consider: Work Order Management is included in a Computerized Maintenance Management System (CMMS) and helps in the scheduling, assignment and tracking of all the facility maintenance activities. The record of these activities can be used to generate vital statistics about facility health, program effectiveness and resource utilization. A Document Management System (DMS) is used to provide electronic storage and retrieval of important facility documentation. This ensures that the material is organized, accessible, backed up and available for auditing. The DMS contains workflow used for document review and approval by Quality Control. 9
- 11. Value-added Innovations Describe the cost management and value-added programs you offer, such as cost reduction programs, energy efficiency programs, and facility improvement programs. Cost Management / Cost Savings Cost Management: How does you manage overall company costs and ensure that reliability is maintained? Cost Reduction Programs: What programs do you implement in order to reduce facility costs? Are cost savings programs structured or unstructured? Are they considered in-scope or out of scope? Provide past examples. Contract Negotiation: Are you able and willing to negotiate discounts for maintenance or other third party contracts? Is the vendor willing/able to take some activities in-house to reduce overall costs? Provide past examples. Lowering TCO: How does your overall program assist in lowering our total cost of ownership while maintaining or even improving facility reliability? Soft Services: Do you provide soft services (custodial, grounds upkeep, security, etc) in addition to critical facility support/maintenance functions? If so, how are these broken out? Value Added Services: What other value added services do you feature and have implemented for your clients? How would these services be beneficial to our critical facilities? Efficiency Improvements (Energy Efficiency) Describe in detail your approach to operational energy efficiency. Efficiency Improvement Programs: What efficiency improvement programs do you offer? How is it structured within the contract? Describe your overall approach and detail steps towards optimization. Provide detailed general examples as well as past client examples. Automation / Monitoring tools: Do you incorporate automation and monitoring tools and systems? If so, how? Utility Relations: How do you interact with and leverage utility incentives and relationships to improve efficiency? Industry Groups: What interaction does the vendor have with relevant industry associations, such as LEED, Energy Star and Green Grid? Are they members? Do they participate in or have access to educational resources? Certified personnel? Quality & Process Improvements Describe and provide examples of other value-added programs you offer, such as quality and process improvement programs. 10
- 12. Customer Service / Satisfaction How do you ensure customer satisfaction? Include a description of your processes, goals and philosophies. Existing Client Satisfaction: In what ways do you assess and ensure client satisfaction? How often are these assessments done? Please describe your overall approach, process and philosophies as it relates to client satisfaction? How do you handle complaints/defective work? What tools and procedures are do you use to address client complaints and substandard work? Reliability record: If possible, provide the reliability record in terms of average uptime percentage obtained for your clients. Measuring Quality of Work: What tools, policies/procedures do you institute to qualify, measure and assure your standard of work? Key Performance Indicators: How do you measure performance? What method and how frequently do you conduct KPI reviews? Service Level Agreement: Define, in detail, your service level support and include examples of legacy service level agreements. Customer Service Features: Describe your overall approach to customer service. Do you have a 24X7 service line or operations center? If not 24X7, what are the particular facility types and hours of operation? What is your average meantime to issue resolution? Please provide examples of escalation procedures for any unresolved issues. Please provide any and all remote tools/capabilities in remote issue resolution. Support Services: Please demonstrate what support services will be offered both initially, and ongoing in the successful management of our facilities. 11
- 13. Organizational Values & Policies Business Continuity: Do you have a documented business continuity/disaster recovery plan? Can this plan be provided to us for review? If you are unwilling to share the entire plan, can you provide a general outline? How often is this plan reviewed and how often is it updated? Insurance: Do you meet the necessary insurance coverage of our organization? Supplier Diversity: Are you a minority or woman business enterprise (MBE/WBE)? If so, please provide certification information. Does your organization have a current supplier diversity program? Do you track annual Tier 1 and Tier 2 supplier spend? Industry Memberships: Please provide a list of any/all industry organizational memberships you have and demonstrate how these affiliations contribute to the stability and successful management of our critical facilities. Labor Policies: Please provide a description of your policies surrounding Health & Safety, Child Labor, Environmental Working Conditions and Discrimination policies. Code of Ethics: Do you have an ethical code of conduct policy? If so, what is the process by which your employees adopt and sign off on these ethics standards? How frequently are employees expected to accept this policy? Conclusion The list of crucial elements for a reliable data center operations program is long. For a program that will deliver a business advantage and low total cost of ownership, the list is even longer. But, it‟s absolutely worth it. We hope this checklist will point you in the right direction to getting the most out of your program in terms of reliability, cost savings and energy savings. Even though this document is already very long, please do not use it as an exhaustive guide. Rather, consider it a starting point. Again, we welcome your feedback and suggested additions to this guide. Please feel free to tell us your suggestions in this very short, five question survey so that we can continually expand and improve this guide. 12
- 14. About Steven Manos Steven Manos, account executive for Lee Technologies, is a thirteen year technology services veteran deeply rooted in every layer of design, deployment, and operations management of large scale mission-critical IT facilities & environments. He has successfully and effectively participated in every layer in the information technology stack from a technical program manager building some of the world‟s largest wide area networks, directing multiple IT and business consulting divisions, managing a $20M systems integrator in Chicago, Illinois and has a significant history of developing key client relationships that have resulted in close to $60M in revenue since 1997. Steven became a member of the Gas Industry Standards Board‟s (GISB) technology team and helped shape the early adoption of security systems necessary for energy companies to safely and securely initiate transactions via the internet in the late 1990‟s. Over the past few years, his role has placed him on the opposite side of the raised floor in the critical facility/data center industry where he has been an integral part of some of the largest facility design/build and operations for various U.S. companies. About Lee Technologies Lee Technologies, a Schneider Electric company, is a U.S. based technology solution provider specializing in large-scale data center solutions and professional services to the North American market. Founded in 1983, Lee Technologies designs, builds, operates, monitors and maintains business-critical facilities for some of the most information-reliant private and public sector organizations in the world, including Coca-Cola, Northrop Grumman, the U.S. Department of Defense, Time Warner, Verizon, Harris Corporation, and many others. Ensuring that customer data is always available, reliable and secure, the Lee Technologies team manages more than 700 locations within the U.S. and has eight locations, including its headquarters in Fairfax, Va., and regional representation in Atlanta, Chicago, Dallas, Denver, Houston, Los Angeles, and Seattle. About Schneider Electric As a global specialist in energy management with operations in more than 100 countries, Schneider Electric offers integrated solutions across multiple market segments, including leadership positions in energy and infrastructure, industrial processes, building automation, and data centres/networks, as well as a broad presence in residential applications. Focused on making energy safe, reliable, and efficient, the company's 110,000 plus employees achieved sales of 19.6 billion euros in 2010, through an active commitment to help individuals and organizations "Make the most of their energy." www.schneider-electric.com © 2011 Lee Technologies, Inc. All Rights Reserved 13