10.aws system management
- 1. AWS SYSTEMS MANAGEMENT AND MONITORING OPTIONS DR.RAJAPRAVEEN
- 3. AWS SYSTEMS MANAGER FEATURES • AWS Systems Manager allows you to centralize operational data from multiple AWS services and automate tasks across your AWS resources. You can create logical groups of resources such as applications, different layers of an application stack, or production versus development environments. With Systems Manager, you can select a resource group and view its recent API activity, resource configuration changes, related notifications, operational alerts, software inventory, and patch compliance status. You can also take action on each resource group depending on your operational needs. Systems Manager provides a central place to view and manage your AWS resources, so you can have complete visibility and control over your operations.
- 4. EXPLORER • AWS Systems Manager Explorer is a customizable dashboard, providing key insights and analysis into the operational health and performance of your AWS environment. Explorer aggregates operational data from across AWS accounts and AWS Regions to help you prioritize and identify where action may be required.
- 5. OPSCENTER • OpsCenter provides a central location where operations engineers and IT professionals can view, investigate, and resolve operational issues related to any AWS resource. OpsCenter aggregates and standardizes operational issues, referred to as OpsItems, while providing contextually relevant data that helps with diagnosis and remediation. Engineers working on an OpsItem get access to information such as:
- 6. OPSCENTER Event, resource and account details Past OpsItems with similar characteristics Related AWS Config changes AWS CloudTrail logs Amazon CloudWatch alarms Stack information Other quick-links to access logs and metrics
- 7. OPSCENTER List of runbooks and recommended runbooks Other information passed to OpsCenter through AWS services • This information helps engineers to investigate and remediate operational issues faster. Engineers can use OpsCenter to view and address issues using the Systems Manager console or via the Systems Manager OpsCenter APIs.
- 8. APPLICATION MANAGER • AWS Systems Manager Application Manager helps you investigate and remediate issues with your AWS resources in the context of your applications. With Application Manager, you can discover and/or define your application components, view operations data (e.g. deployment status, Amazon CloudWatch alarms, resource configurations, and operational issues) in the context of an application, and perform remedial actions such as patching and running Automation runbooks. This streamlines operational workflows for your applications, avoiding the need to use different consoles to investigate and remediate operational issues. In addition, you can use Application Manager to view operational data and CloudWatch alarms and take action on your existing container clusters in Amazon EKS environments.
- 9. APPCONFIG • AWS AppConfig helps you deploy application configuration in a managed and a monitored way just like code deployments, but without the need to deploy the code if a configuration value changes. AWS AppConfig scales with your infrastructure so you can deploy configurations to any number of Amazon EC2 instances, containers, AWS Lambda functions, mobile apps, IoT devices or on-premises instances. AWS AppConfig enables you to update configurations by entering changes through the API or Console. AWS AppConfig allows you to validate those changes semantically and syntactically to ensure configurations are aligned to their respective applications’ expectation, thus enabling you to help prevent potential outages. You can deploy your application configurations with similar best practices as code deployments, including staging roll-outs, monitoring alarms, and roll back changes should an error occur.
- 10. CHANGE MANAGER • AWS Systems Manager Change Manager simplifies the way you request, approve, implement, and report on operational changes to your application configuration and infrastructure on AWS and on-premises. With Change Manager, you can use pre-approved change workflows to help avoid unintentional results when making operational changes. Change Manager helps you safely implement changes, while detecting schedule conflicts with important business events and automatically notifying impacted approvers. Using Change Manager’s change reports, you can monitor progress and audit operational changes across your organization, providing improved visibility and accountability.
- 11. AUTOMATION • AWS Systems Manager allows you to safely automate common and repetitive IT operations and management tasks. With Systems Manager Automation, you can use predefined playbooks, or you can build, run, and share wiki-style automated playbooks to enable AWS resource management across multiple accounts and AWS Regions. You can execute Python or PowerShell scripts as part of a playbook in combination with other automation actions such as approvals, AWS API calls, or running commands on your EC2 instances. These playbooks can be scheduled in a maintenance window, triggered based on changes to AWS resources through Amazon CloudWatch Events, or executed directly through the AWS Management Console, CLIs, and SDKs. You can track the execution of each step in a playbook, require approvals, incrementally roll out changes, and automatically halt the roll out if errors occur.
- 12. MAINTENANCE WINDOWS • AWS Systems Manager lets you schedule windows of time to run administrative and maintenance tasks across your instances. This ensures that you can select a convenient and safe time to install patches and updates or make other configuration changes, improving the availability and reliability of your services and applications.
- 13. FLEET MANAGER • AWS Systems Manager Fleet Manager streamlines your remote server management process. With Fleet Manager, you can easily manage and troubleshoot your fleet of servers running on AWS and on-premises. You can drill down to individual servers to perform common troubleshooting and management tasks using a centralized graphical user interface. Perform a variety of system administration tasks, including disk and file exploration, log management, Windows Registry operations, and user management, without needing to remotely connect to your virtual machines, saving your administrators time and effort.
- 14. COMPLIANCE • AWS Systems Manager automatically aggregates and displays operational data for each resource group through a dashboard. Systems Manager eliminates the need for you to navigate across multiple AWS consoles to view your operational data. With Systems Manager you can view API call logs from AWS CloudTrail, resource configuration changes from AWS Config, software inventory, and patch compliance status by resource group. You can also easily integrate your AWS CloudWatch Dashboards, AWS Trusted Advisor notifications, and AWS Personal Health Dashboard performance and availability alerts into your Systems Manager dashboard. Systems Manager centralizes all relevant operational data, so you can have a clear view of your infrastructure compliance and performance.
- 16. INVENTORY • AWS Systems Manager collects information about your instances and the software installed on them, helping you to understand your system configurations and installed applications. You can collect data about applications, files, network configurations, Windows services, registries, server roles, updates, and any other system properties. The gathered data enables you to manage application assets, track licenses, monitor file integrity, discover applications not installed by a traditional installer, and more.
- 18. SESSION MANAGER • AWS Systems Manager provides a browser-based interactive shell and CLI for managing Windows and Linux EC2 instances, without the need to open inbound ports, manage SSH keys, or use bastion hosts. Administrators can grant and revoke access to instances through a central location by using AWS Identity and Access Management (IAM) policies. This allows you to control which users can access each instance, including the option to provide non-root access to specified users. Once access is provided, you can audit which user accessed an instance and log each command to Amazon S3 or Amazon Cloud Watch Logs using AWS CloudTrail.
- 19. RUN COMMAND • AWS Systems Manager provides you safe, secure remote management of your instances at scale without logging into your servers, replacing the need for bastion hosts, SSH, or remote PowerShell. It provides a simple way of automating common administrative tasks across groups of instances such as registry edits, user management, and software and patch installations. Through integration with AWS Identity and Access Management (IAM), you can apply granular permissions to control the actions users can perform on instances. All actions taken with Systems Manager are recorded by AWS CloudTrail, allowing you to audit changes throughout your environment.
- 20. STATE MANAGER • AWS Systems Manager provides configuration management, which helps you maintain consistent configuration of your Amazon EC2 or on-premises instances. With Systems Manager, you can control configuration details such as server configurations, anti-virus definitions, firewall settings, and more. You can define configuration policies for your servers through the AWS Management Console or use existing scripts, PowerShell modules, or Ansible playbooks directly from GitHub or Amazon S3 buckets. Systems Manager automatically applies your configurations across your instances at a time and frequency that you define. You can query Systems Manager at any time to view the status of your instance configurations, giving you on-demand visibility into your compliance status.
- 21. PATCH MANAGER • AWS Systems Manager helps you select and deploy operating system and software patches automatically across large groups of Amazon EC2 or on-premises instances. Through patch baselines, you can set rules to auto-approve select categories of patches to be installed, such as operating system or high severity patches, and you can specify a list of patches that override these rules and are automatically approved or rejected. You can also schedule maintenance windows for your patches so that they are only applied during preset times. Systems Manager helps ensure that your software is up-to-date and meets your compliance policies.
- 22. DISTRIBUTOR • AWS Systems Manager helps you securely distribute and install software packages, such as software agents. Systems Manager Distributor allows you to centrally store and systematically distribute software packages while you maintain control over versioning. You can use Distributor to create and distribute software packages and then install them using Systems Manager Run Command and State Manager. Distributor can also use Identity and Access Management (IAM) policies to control who can create or update packages in your account. You can use the existing IAM policy support for Systems Manager Run Command and State Manager to define who can install packages on your hosts.
- 23. CONNECT WITH ITSM / ITOM SOFTWARE • IT Service Management (ITSM) tools, such as Jira Service Desk, can connect with AWS Systems Manager to make it easier for ITSM platform users to manage AWS resources. These AWS Service Management Connectors provide Jira Service Desk administrators governance and oversight over AWS products.
- 24. WHAT IS AWS MONITORING? • Amazon Web Services (AWS) monitoring is a set of practices you can use to verify the security and performance of your AWS resources and data. These practices rely on various tools and services to collect, analyze, and present data insights. You can then use these insights to identify vulnerabilities and issues, predict performance, and optimize configurations.
- 25. AWS FIRST-PARTY MONITORING TOOLS • There are multiple services and utilities available from AWS that you can use to monitor your systems and access. Some of these tools are included in existing services, while others are available for additional costs.
- 26. AWS CLOUDTRAIL • CloudTrail is a service that you can use to track events across your account. The service automatically records event logs and activity logs for your services and stores the data in S3. Collected data includes user identities, traffic origin IPs, and timestamps. You can view all management events for free for the most recent 90 days. Data events and insights based on your data are also available for an additional fee.
- 27. AWS CLOUDWATCH • CloudWatch is a service you can use to aggregate, visualize, and respond to service metrics. CloudWatch has two main components: alarms, which create alerts according to thresholds for single metrics, and events, which can automate responses to metric values or system changes.
- 28. AWS CERTIFICATE MANAGER • Certificate Manager is a tool you can use to provision, manage, and apply transport layer security (TLS) and secure sockets layer (SSL) certificates. These certificates are used to prove your services or devices' authenticity and enable you to secure network connections.
- 29. AMAZON EC2 DASHBOARD • EC2 Dashboard is a monitoring tool for the Amazon EC2 virtual machine service. You can use this dashboard to monitor and maintain your EC2 instances and infrastructure. The dashboard lets you view instance states and service health, manage alarms and status reports, view scheduled events, and assess volume and instance metrics
- 30. AWS THIRD-PARTY MONITORING TOOLS • In addition to native tools, many AWS users also adopt third-party tools. These tools are useful for separating monitoring operations from your primary resources and can often provide support for hybrid or on-premises resources as well.
- 31. NETAPP CLOUD INSIGHTS • NetApp Cloud Insights is a tool for monitoring that you can use to visualize your infrastructure.It enables you to monitor, optimize, and troubleshoot resources in public and private clouds and on-premises. Cloud Insights includes features for conditional alerting, optimization recommendations, predictive analytics, machine learning based anomaly detection, and compliance auditing.
- 32. SOLARWINDS APPOPTICS • AppOptics is a tool that you can use to supplement metrics collected by CloudWatch. It enables you to track performance statistics, log trends, and capacity limits. You can integrate AppOptics with other AWS services and generate automatic analyses of your operations. AppOptics also includes features that enable you to monitor multiple AWS accounts from a single interface.
- 33. ZENOSS ZENPACK • ZenPack is an open source tool you can use to aggregate CloudWatch metrics and external resource metrics data. It includes an easy to use graphical user interface (GUI) and is compatible with a variety of AWS services. These services include S3, Amazon Virtual Private Cloud (VPC), and Amazon Suite.
- 34. • Zabbix • Zabbix is an open source tool for collecting metrics from AWS and a variety of other applications, services, and databases. It includes features for dashboards, alert escalation, and a robust online community of support. The downside of Zabbix is that it cannot import data or generate performance reports. • Weave Scope • Weave Scope is an open source tool you can use to monitor and visualize your microservices. It includes features for service discovery and is compatible with Elastic Container Services (ECS). Weave Scope is based on three components (an interface, an app, and a probe) and enables you to troubleshoot service performance in real time.
- 35. STEPS FOR SUCCESSFULAWS RESOURCE MONITORING • Phase A: Assess Your AWS Monitoring Needs • Before introducing monitoring into your pipeline or making changes to your existing workflow, you should carefully assess your existing infrastructure, tooling, resources, and skillset. Taking the time to assess your situation can help you develop a strategy that suits your needs.
- 36. STEP 1: ASSESSMENT QUESTIONS • Here are key questions to ask when assessing your AWS monitoring needs: Infrastructure—where is your network located? Is it on-premise? Do you want a dedicated monitoring system for each environment or do you want to do on-premise monitoring with cloud monitoring using one tool? Compliance—what are your current compliance policies? What legal percussions do you need to take in order to comply with industry standards? Can you introduce a SaaS monitoring and logging solution into your ecosystem and remain compliant? Inventory—do you need a new tool for AWS monitoring or can your current stack perform this task? Complexity—what are the complexities and costs involved in removing any and all legacy agents from all servers, to clear space for the installation of new agents? Metrics—do you know which metrics you absolutely need to monitor, and which metrics might be redundant?
- 37. STEP 2: DEVELOP A STRATEGY TO TAG AWS RESOURCES Once you gain insight into your current monitoring needs and prioritize metrics, you can start developing a strategy for tagging AWS resources. Tags help you keep track of your resources, and monitor usage and behavior. • If you don’t have a tagging system in place, it can take some time to figure out how to organize resources. While every project and organization is unique, it is important to create a tagging system that can be used by a wide variety of professionals and collaborators. This way, all relevant parties can gain access to monitoring insights when needed.
- 38. PHASE B: SELECT THE RIGHT SOLUTION FOR YOUR ORGANIZATION • After assessing your needs and setting up a tagging system for AWS resources, you can look for the solution that suits your needs. Often, it is effective to start with a simple solution and then expand as needed. However, if you know in advance you need a robust set of features, it’s best to go with a solution that either fits your needs, can be scaled easily, or meets all criteria.
- 39. STEP 3: START SIMPLE WITH AMAZON CLOUDWATCH • CloudWatch metrics can help you monitor practically any AWS resource. CloudWatch provides a wide range of pre-built counters like DiskQueueLength and CPUUtilization. Some AWS services, such as RDS and EC2, can provide additional counters when integrated with CloudWatch. • CloudWatch counters enable you to create dashboards, which you can leverage when you need visualized data. In addition to counters and dashboards, CloudWatch offers an alerting system, which lets you know when incidents occur. If you are not using a dedicated monitoring system, and you need simple features, you can use CloudWatch.
- 40. STEP 4: LEVERAGE BEST-OF-BREED SOLUTIONS • When it comes to visibility, the more resource types you monitor, the more you can ensure the performance and safety of your assets. However, not all monitoring systems can provide visibility for all resources. Some monitoring solutions are designed for infrastructure while others are built for network traffic. • To avoid losing visibility over parts of your environment, you can either use a stack of tools or you can extend the capabilities of existing systems. If you opt to use a stack of monitoring, you might want to first check that the tools provide the features you require and are compatible with each other and your existing stack.
- 41. STEP 4: LEVERAGE BEST-OF-BREED SOLUTIONS • Additionally, you should consider adding a tool to centralize the stack, to ensure productivity remains effective. If you choose to extend existing systems by installing plugins or integrating with APIs, you should enable AWS integration and ensure that each extension is compliant with any regulatory requirements you are legally required to uphold.
- 42. PHASE C: CAPTURE LOGS • Once you set up your monitoring solution or stack, you should decide which logs you want to capture and how you want to set this up. Logs are highly effective for keeping track of compliance requirements and troubleshooting issues.
- 43. PHASE C: CAPTURE LOGS • Here is a list of logs you might want to capture: Database logs—help you detect queries that are slow to run. Application logs—point out application failures. AWS CloudTrail—detects API calls made to AWS. Elastic Load Balancing and host logs—might indicate availability or latency changes. OS logs—can identify host failure reasons. Web server logs—as well as firewall logs and VPC flow logs can detect patterns of access and attacks. • The majority of monitoring systems are either suited for metrics or logs, rather than prioritizing both of these tasks equally. To ensure full coverage, you should either use a stack or find a solution that enables you to capture both metrics and logs from AWS.
- 44. AWS MONITORING BEST PRACTICES • When monitoring your AWS resources, the following best practices can help you ensure that no resources are overlooked and that you can troubleshoot efficiently.
- 45. USE AUTOMATION WHERE POSSIBLE • Production deployments in AWS are typically too large and dynamic to monitor manually. The volume of metrics and log data that is generated is too large for humans to efficiently analyze. To ensure that critical data is not missed and responses are timely, you should use automation to handle most of your monitoring tasks.
- 46. CREATE POLICIES TO DEFINE PRIORITY LEVELS • Prioritizing monitoring tasks helps ensure that critical services remain operational and that data remains protected. Additionally, prioritizing alerts or alert categories helps ensure that IT teams effectively distribute their time and efforts.
- 47. RESOLVE PROBLEMS EARLY ON • Monitoring data should be used to respond to issues like potential service interruptions proactively. It is much easier to scale resources or throttle traffic in advance than manage a service outage. Additionally, addressing potential issues early on can help you avoid wasted resources and costs.
- 48. USE THE CLOUD TO YOUR ADVANTAGE • Cloud environments are flexible and can enable you to experiment with configuration changes without affecting services. When optimizing based on metrics, take time to test your configurations. This way, you can verify if changes are more efficient before implementing them in production.
- 49. AWS MONITORING WITH NETAPP CLOUD INSIGHTS • NetApp Cloud Insights is an infrastructure monitoring tool that gives you visibility into your complete infrastructure. With Cloud Insights, you can monitor, troubleshoot and optimize all your resources including your public clouds and your private data centers. • Cloud Insights helps you find problems fast before they impact your business. Optimize usage so you can defer spend, do more with your limited budgets, detect ransomware attacks before it’s too late and easily report on data access for security compliance auditing. • In particular, NetApp Cloud Insights lets you automatically build topologies, correlate metrics, detect greedy or degraded resources, and alert on anomalous user behavior
- 50. AWS MONITORING BEST PRACTICES • Monitoring cloud environments can be quite different than on-premises ones. These environments are dynamic, highly distributed, and inherently more vulnerable to cyber threats. To ensure that you are applying the proper strategies when monitoring your cloud resources, it is important to make sure you are following best practices. • This article explains what AWS monitoring best practices are, how monitoring in AWS works, and highlights 6 best practices for ensuring effective monitoring in AWS.
- 51. AWS MONITORING DASHBOARD • Dashboards are an effective way to centralize your metrics monitoring and provide information to teams quickly. You can use these tools to ensure that your entire team is working from reliable information or to share the status of your operations with executive and shareholders. • This article explains what AWS monitoring dashboards are, the components of a dashboard, provides two tutorials for creating dashboards, and highlights some best practices.
- 52. CLOUDWATCH MONITORING • Monitoring your AWS resources is one of the best ways to ensure that your services and applications remain performant and cost effective. To make this monitoring easier, AWS offers a service called CloudWatch which you can use to collect and visualize metrics across your services. • This article explains what CloudWatch monitoring is, how CloudWatch works, some key concepts to know in CloudWatch, and highlights a few metrics to watch for EBS and EC2.
- 53. CLOUDWATCH LOG INSIGHTS • The ability to query and interpret logs enables you to derive greater insights from your data. It also enables you to diagnose and identify issues or opportunities for improvement faster. In AWS, you can use CloudWatch Logs Insights to perform these tasks and ensure your operations continue smoothly. • This article explains what CloudWatch Logs Insights is, how to get log data to the service, what the syntax for queries is, and how to perform a sample query.
- 54. MONITORING THE COSTS OF UNDERUTILIZED EBS VOLUMES • Overprovisioning your resources can eat away at your carefully planned cloud resource budgets and limit the amount of value you gain from services. To prevent wasted costs, it’s important to make sure that your resources are right-sized for your operations and are being used efficiently. • In this article you’ll learn how to find underperforming resources in EBS, how to evaluate your resource use, and how to apply metrics to improve your resource efficiency.