How to Design, Build and Map IT and Business Services in Splunk

Editor's Notes

• #2: Good afternoon folks. Thank you very much for joining us today. <intro to self> Questions to group SL Attendance Attendee titles Splunk expertise Use case
• #3: Who’s been doing Splunk for more than 1 yr Pony or 2 fro those who’ve been here for long Understand the audience: No Ponies: Identify the people who map to LOB,s PMs, service owners – so we can align with them
• #5: At the end of this session, there are 3 key things I’d like you to walk away with. First, there’s been a lot of talk from Splunk on Service Intelligence. So, I’d like to define that and of what we mean by that and why it matters. Once we have that baseline, we can then, go into how to derive service Intelligence. How do you identify the right measurements, and map those success metrics to data sets that will contribute to these measurements. Lastly, I’d like you to introduce you to a new offering from Splunk that has made tremendous headway and one that our customers are lvoing love called Splunk IT Service Intelligence (ITSI). And how you can use ITSI to map these measurements to deliver real-time service insights.
• #6: At Splunk, our mission is to make machine data accessible, usable and valuable to everyone. And this overarching mission is what drives our company and product priorities.
• #7: Unlike traditional structured data or multi-dimensional data– for example data stored in a traditional relational database for batch reporting – machine data is non-standard, highly diverse, dynamic and high volume. You will notice that machine data events are also typically time-stamped – it is time-series data.   Take the example of purchasing a product on your tablet or smartphone: the purchase transaction fails, you call the call center and then tweet about your experience. All these events are captured - as they occur - in the machine data generated by the different systems supporting these different interactions.   Each of the underlying systems can generate millions of machine data events daily. Here we see small excerpts from just some of them.
• #8: If you can correlate and visualize related events across these disparate sources, you can build a picture of activity, behavior and experience. And what if you can do all of this in real-time? You can respond more quickly to events that matter. For example, if an organizations captured the customers twitter ID in their customer profile this correlation would be possible. Where that didn’t exist, they could at least group by demographic with the tweets. You can extrapolate this example to a wide range of use cases – security and fraud, transaction monitoring and analysis, web analytics, IT operations and so on.
• #9: Splunk products are being used for data volumes ranging from gigabytes to hundreds of terabytes per day. Splunk software and cloud services reliably collects and indexes machine data, from a single source to tens of thousands of sources. All in real time. Once data is in Splunk Enterprise, you can search, analyze, report on and share insights form your data. The Splunk Enterprise platform is optimized for real-time, low-latency and interactivity, making it easy to explore, analyze and visualize your data. This is described as Operational Intelligence. The insights gained from machine data support a number of use cases and can drive value across your organization. [In North America] Splunk Cloud is available in North America and offers Splunk Enterprise as a cloud-based service – essentially empowering you with Operational Intelligence without any operational effort.
• #10: Splunk products are being used for data volumes ranging from gigabytes to hundreds of terabytes per day. Splunk software and cloud services reliably collects and indexes machine data, from a single source to tens of thousands of sources. All in real time. Once data is in Splunk Enterprise, you can search, analyze, report on and share insights form your data. The Splunk Enterprise platform is optimized for real-time, low-latency and interactivity, making it easy to explore, analyze and visualize your data. This is described as Operational Intelligence. The insights gained from machine data support a number of use cases and can drive value across your organization. [In North America] Splunk Cloud is available in North America and offers Splunk Enterprise as a cloud-based service – essentially empowering you with Operational Intelligence without any operational effort.
• #11: So, let’s jump right into what Service Intelligence is. If you look at most enterprises today, IT is just not supporting the business, but is being defined by the software and services supported by IT. For example, if you’re depositing a check using your bank’s mobile app, or if you’re streaming a video online, or if you’re purchasing something online – it’s all software and systems, supported by IT. IT has moved to the forefront of the business, responsible for the health of not just its operations, but also the business. However, if you look at how IT operates in large enterprises, they continue to struggle to stay relevant, because they’re kind of stuck in a rock and a hard place. On one side they are being bombarded by the business to improve availability, performance and user experience but at the same time, they are trapped by legacy tools and operational silos that are not ideal to solve modern day problems. So, what you really need is be intelligent about the services IT delivers and understand how exactly these services relate to business goals and objectives Be intelligent when an issue arises. Understand it more holistically – and by that I mean, not just have an end-to-end picture of services and it’s underlying component parts, but also understand how to break down the problem methodically. Prioritize issues, collaborate across silos with a single, yet customized view of the services and sub-services, and organize and correlate issues to be able to triage the problem and resolve it quickly And lastly, just be intelligent – you have data, an abundance of it. You just need to enable yourselves with intelligently using that data to solve emerging problems with analytics-driven insights – so much like how we hold holds across operations when a problem occurs over a call, how do we enable the machine to do the high-volume drudge work and enable you to exercise the right judgment and high-volume reasoning. Sounds like a lot of work and something that’s been promised for decades – yes? How many of you went down this path in some sort and then gave up coz it was insanely hard to get there? Well, I agree with you that this sounds daunting. I come from that world myself. I worked for Mercury Interactive in early 2000’s when we had embarked on this road, and then we got acquired and while many of the enterprise vendors touted to do this, they couldn’t. I can go into many reasons on why it didn’t work – like normalized databases, CMDBs, tools not built for dynamic environments, but I’m not here to tell you why that didn’t work. I’m here to tell you that this is now very achievable. There is a method to this madness – trust me and it’s very achievable. Here are some examples of what our customers are doing.
• #12: If you look at this customer, Converging Data, they have a very customized visualization of their service – in just a single view, they are able to get overarching insights into their healthcare delivery scorecard, what the tertiary services like pathology, facilities, patient transportation and so on looks like, what their specialized services in maternity, NICU, Icu are looking like and so on. This was achieved with Converging Data in 1 single day.
• #13: Here’s another example This customer is a health care insurance service provider too, but one of their most critical services was claims process – providing the ability for their doctors and insurance agencies to submit claims, so that doctors can get paid for the services they have provided to their customers. Here, you can quickly see how many claims are being supported, what the health of their supporting infrastructure is looking like and the health of this infrastructure, and service and you can also see a mapping of what this means to the business, in terms of how many claims they are processing in real-time and what the monetary impact is. This is what our customers are doing with it - they are able to look at the data not just from an IT perspective, but also from business volumes. The technology is there, you just need to leverage it. Just yesterday, I was doing an interview for a customer their VP and AVP of infrastructure services – they’re presenting for us Gartner Datacenter and they mentioned that these kinds of visualizations are so powerful, coz often times the business wants to see the overall picture of the environment without too many technical details and these kinds of dashboards are powerful, intuitive and easy to understand while still maintaining to drill down logically into the component parts for deeper insights.
• #14: So, how do you go about solving this problem. Let’s take buttercup games, because of course, buttercup is the Splunk mascot. Buttercup Games manufactures stuffed toys and games. Let’s figure out how to uncover the services important to the company and where there are problems worth solving. As a manufacturing company, the supply chain is extremely important. It’s a system that allows us to track the flow of good. Additionally, they have embarked on a new digital experience for their users with a new online store and want to make sure that their customers and the business are both not impacted.
• #15: Let’s break that down further. First, we spoke about supply chain. The various aspects of supply chain would include order entry, manufacturing, shipping, fulfillment and you need a way to track and streamline these operations to minimize disruptions Second, the supporting pieces to this supply chain would include ERP systems and processes that ensure the business process functions without any hitches, so how do you measure it’s efficiency Next, we have the online store – without which the customers are unable to place orders. There is a direct impact to the top-line and therefore customer experience is of supreme importance Operationally, how do you ensure that things are going well? When customer care says that orders are failing and customers are complaining, how do you avoid spending 32 hours / week troubleshooting and triaging the issues in a war room? And lastly, what does this mean in terms of revenue impac – are you able to identify it and quantify it and plant a number on impact to revenue? So, we started with – we know that buttercup games is an toy store manufacturer, needs insights into supply chain, and lastly, insights into their online store (operationally and business impact).
• #16: Now, how do you break this highlevel problem into something that’s understandable by all parties, to derive service intelligence, like we defined. A recap of what is Service Intelligence – 1 – ability to measure IT services to business impact 2 – ability to unlock operational efficiencies and 3 – drive data based decisions First, start with a problem worth solving. This seems obvious, but unfortunately, does not always happen. We often get so busy doing our day to day job that we don’t or are unable to take the time to step back and look at what matters Second, now that you have identified a problem that needs resolution, bring the right team together to solve the problem. Like xxxx Lastly, design what you want to say. 80% of the problem is figuring out what you want and why you want it. Once you have that laid out then, getting it done is the easier part. Let me elaborate on each of these a bit more.
• #17: The most effective approach to building a model usually starts with identifying a business opportunity and determining how the model can improve performance and outcomes. Solving a business problem is thinking beyond the symptom or finding and fixing “issues.” What are the top business services in your enterprise? How do you measure the customer experience with these services? Are customers happy with their experience?
• #18: It’s about understanding how the delivered services impact operations and the business. This can be accomplished by creating a comprehensive list of services, noting how they match to the priorities of the business and corresponding SLAs, and determining which services have the biggest business impact. Is it impactful, valuable measurable Drive decision making with quantifiable measurements How do you drive decisions to meet business needs Identify What are the top business services in your enterprise? How do you measure the customer experience with these services? Are customers happy with their experience?
• #19: Let’s go back to buttercup games. Supply chain – business critical problem Measruing the business process and efficiency of the business process – check, business critical Online store – business critical – so a check there How many hours are we spending supporting this app, if we spend about 32 hrs/wk in war rooms? – 32 x 52 = 1664 hrs How does this impact the business if downtime costs the company 48,000 per week - ~$2.5M So, now, you’ve essentially identify the service, the business process and the applications that enable the service. What it means in terms of business impact and how to measure critical indicators that matter to the business How big is the business impact in terms of ours and in terms of revenue loss Pnoy: How many hours are we spending supporting this app - (32 x 52) Pony: How big is the business problem? (48,000 X 52)
• #20: But you can’t do that in a silo. In order to understand what matters to the business and accurately map the service flow end-to-end, you need both the business and technical stakeholders. You want to collaborate on this instead of trying to figure this out in a silo. A collborative approach ensures optimized outcomes, alignment in prioritizing services, and the ability to record key measurements to construct an accurate service map. How often do customers experience issues with the service? When issues arise, who gets involved in resolving them? How do teams work together to resolve issues? Evaluate the performance of a process or a service – the measurements can be based upon the effectiveness (business value derived) or efficiency (how quickly the service is delivered) Identify pains, performance indicators and measurement goals for the service Develop an end-to-end map of the services
• #21: Some of the folks you want to consider are: Service owners: These are the folks that speak business. They have an overall understanding of the service but do not necessarily know or care about the underlying supporting tiers. What they bring to the table is insight into the indicators that need to be measured that are meaningful to the business Operations and support folks are the more hands-on folks. They are the first line of defense for the company. They have a completely different view to the same service than what service owners do, as they have diffeernt KPIs they are measured against. This could include MTTR, FCR, Frequency of issues and just the inner workings of the systems and infrastructure. Enterprise architects are really important in the disussion coz they truly understand ow things really work. In one organization we had visited recently, we had an argument between the architects and support teams, each claiming the inner workings of the service and in some ways they were both right. The architects are the ones that design the system but the operations folks are the ones that help support it on an ongoing basis. So, both are very critical in this discussion And lastly you have the administrators – who use the tools. For instance, you will include your Splunk experts – these are the guys that can tell you whether or not you have the right data to correctly map and measure what you need. What tools are in place and what kind of infomritaon can you actually get into the system for accurately measruing the critical KPIs identified by these stakeholders Titles for monitoring tools manager Put people pictures next to them (map to Marc Olesen, Ravi Anandawalla, John Butler) Enteprise architects are really important coz they understand how things REALLY work. Once architects and support team got into an argument and
• #23: The last step in the process is to design. Organizations often articulate priorities using business terminology, such as “improved customer service” or “reduced time-to-market.” To make sense of this, IT stakeholders must break down the organization’s value chain into its constituent technologies—what are the applications, services, hosts and devices that make up the value chain. IT operators must be able to create KPIs that turn data into meaningful information that the business can understand. This can be accomplished by sourcing the precise data and defining interactive service topology maps, to provide an overall perspective on the health of your critical services and supporting infrastructure. What components do we need to include in the service; db, middleware What data is needed to drive the metrics Meet with business leaders, and their teams, to review the consolidated mapping and modify as necessary
• #25: So, why did the old ways fail – I’ll go over this very quickly. When datacenters were originally built, they were built in silos and tools supporting insights into the datacenter was also built to support these silos. However, let’s look at a traditional service stack – it can be broken down into 4 layers - the infrastructure layer, application layer, business layer and then finally the service layer. And now you had various tools supporting each of these layers. The way single- source-of-truth” service management solutions attempted to manage service health was by ‘poorly’ integrating these various tools with connectors and products with fixed, relational stores that required herculean yet fragile custom interconnects, and they failed. Filtering and normalizing data to fit these predefined constructs often left IT with shallow business- and IT-service-centric views. These views lacked depth in their understanding of root cause, service levels and business impact. And with today’s IT environments being dynamic and no longer contained within the walls of the enterprise, this problem was further exaggerated and became more intense.
• #27: This isn’t hard at all. With these 10 data sources, you can do the full design
• #29: With Splunk ITSI, customers get the higher level benefits based on the underlying platform. So, from deep-in-the-weeds solving IT operational use cases with Splunk enterprise, we’re up-leveling the use cases and making IT more relevant to the business. The can visualize meaningful and contextual data and inter-relationships with dynamic service models, organize and correlate performance indicators for at-a-glance problem analysis, get proactive with early warnings on anomalies, deviations and pre-configured correlated alerts, and simplify workflows.
• #34: Bring up live system