Creating a Self-Healing Mechanism for a Lightweight Website

This article was written by Nestor Mayagma Jr. Nestor is a cloud engineer and member of the AWS Community Builder. He continuously strives to expand his knowledge and expertise in AWS to foster personal and professional growth. He also shares his insights with the community through numerous AWS blogs, highlighting his commitment to Cloud Computing technology. In his leisure time, he indulges in playing FPS and other online games.

Website downtime can cause significant disruption, but for lightweight sites such as personal blogs or static websites, a quick instance restart may be enough to fix the issue. This article will guide you through creating a self-healing mechanism with AWS Lambda and Amazon EventBridge. This solution will automatically identify when your site goes down, restart it, and notify you via Slack. Additionally, we’ll show how Amazon CloudWatch can trigger the Lambda function using a subscription filter, and we’ll cover common causes of downtime or 5xx errors.

Why Self-Healing?

The internet's unpredictability can sometimes cause website downtime, impacting your visitors' experience. Websites may go offline for various reasons, such as server overload, resource depletion, or network problems. A self-healing mechanism can automatically detect when your site is down and take corrective actions, like restarting the instance, without needing manual input. This process helps reduce downtime, ensuring your website stays accessible and offering a more reliable and smooth user experience.

Amazon Lightsail is a straightforward service offering virtual private servers (instances) with a predictable pricing structure, making it perfect for lightweight websites such as personal blogs or static sites. While Lightsail instances are generally dependable, there may be times when a restart is needed to resolve temporary problems. By leveraging AWS Lambda and EventBridge, we can create an automated self-healing mechanism that monitors and restarts Lightsail instances under certain conditions. This solution streamlines management and improves the overall stability of your website.

Note: This solution can also be applied to Amazon EC2 instances. With a few adjustments to the Lambda function and permissions, you can enable similar self-healing functionality for EC2 instances, offering greater flexibility in managing your cloud infrastructure.

Causes of Downtime

Typical causes for downtime or 5xx errors include:

• Server Overload: Excessive requests can overload the server.
• Resource Depletion: Lack of adequate memory, CPU, or disk space.
• Configuration Errors: Incorrect server or application settings.
• Network Issues: Issues with network connectivity or DNS.
• Application Bugs: Bugs in the code or issues with dependencies.

Benefits of a Self-Healing Mechanism

• Minimized Downtime: Automated recovery actions shorten the duration of downtime.
• Reduced Manual Intervention: Automation removes the requirement for continuous monitoring.
• Improved User Experience: Guarantees the website stays accessible and operational.

Setup Overview

In this example, we will be using:

• AWS Lambda: To execute the self-healing script.
• Amazon EventBridge: To initiate the Lambda function.
• Amazon Lightsail: To host the website (which can be replaced with Amazon EC2 if preferred).
• Slack: To receive notifications.

Here’s a quick summary of the process:

1. Lambda Function: This function verifies if the website is accessible. If it's not, it restarts the Lightsail instance and sends a notification to Slack.
2. EventBridge Rule: This rule activates the Lambda function at set intervals (e.g., every 5 minutes).
3. CloudWatch (Optional): CloudWatch can also trigger the Lambda function using specific metrics or logs.

Step-by-Step Guide

1. Create the Lambda Function

Here’s an example of a Lambda function to set up the self-healing mechanism. You can adjust this code according to your needs:

import boto3
import urllib.request
import json
import os

# Initialize AWS Lightsail client
lightsail = boto3.client('lightsail')

# Define instance details
instance_name = 'JR' # Don't forget to replace this
slack_webhook_url = 'https://hooks.slack.com/services/..' # Don't forget to replace this

def check_website(url):
    try:
        response = urllib.request.urlopen(url, timeout=10)
        return response.getcode() == 200
    except urllib.error.URLError:
        return False

def restart_lightsail_instance():
    try:
        print(f"Rebooting Lightsail instance {instance_name}...")
        lightsail.reboot_instance(instanceName=instance_name)
        
        # Wait for the instance to stop
        # waiter = lightsail.get_waiter('instance_stopped')
        # waiter.wait(instanceNames=[instance_name])
        
        # print(f"Starting Lightsail instance {instance_name}...")
        # lightsail.start_instance(instanceName=instance_name)
        print("Instance rebooted successfully.")
    except Exception as e:
        print(f"Failed to restart Lightsail instance: {e}")

def send_slack_notification(message):
    payload = {
        "text": message
    }
    headers = {
        'Content-Type': 'application/json'
    }
    req = urllib.request.Request(
        slack_webhook_url, 
        data=json.dumps(payload).encode('utf-8'),
        headers=headers
    )
    try:
        urllib.request.urlopen(req)
        print("Slack notification sent successfully.")
    except Exception as e:
        print(f"Failed to send Slack notification: {e}")

def lambda_handler(event, context):
    website_url = 'https://nestormayagmajr.com'
    
    if check_website(website_url):
        message = f"{website_url} is reachable. No action needed."
        print(message)
    else:
        message = f"{website_url} is unreachable. Rebooting Lightsail instance."
        print(message)
        restart_lightsail_instance()

    send_slack_notification(message)

# For local testing
if __name__ == "__main__":
    lambda_handler(None, None)        

This Lambda function monitors the website's availability, restarts the Amazon Lightsail instance if the site is down, and sends an alert to Slack.

2. Configure Lambda Permissions

To enable the Lambda function to restart the Lightsail instance, you must set up the appropriate permissions. Below is the required IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lightsail:GetInstanceAccessDetails",                
                "lightsail:GetInstances",                
                "lightsail:RebootInstance",
                "lightsail:GetInstance",
                "lightsail:GetInstanceState"
            ],
            "Resource": "*"
        }
    ]
}        

Attach this policy to the Lambda execution role to provide the required permissions.

3. Create the EventBridge Rule

To set up an EventBridge rule that activates the Lambda function:

1. Navigate to the Amazon EventBridge console.
2. Select Create rule.
3. Set a name and description for the rule.
4. Choose EventBridge schedule as the Event source.
5. Define the schedule (e.g., every 5 minutes).
6. Add a target and choose the Lambda function created earlier.

4. (Optional) CloudWatch as a Trigger

For a real-time solution, you can use Amazon CloudWatch to trigger the Lambda function based on certain metrics or logs. For instance, set up a CloudWatch alarm to monitor HTTP status codes and trigger the Lambda function if a 5xx or any error status code is detected. This involves sending access logs to CloudWatch using the CloudWatch Agent and creating a subscription filter to verify if the site is down.

Subscription Filter in CloudWatch

CloudWatch enables the creation of subscription filters to monitor logs and trigger actions based on particular patterns. For instance, you can use a filter pattern to identify 5xx status codes in access logs:

[ip, identity, user, timestamp, request, statusCode=5*, size, userAgent]

This pattern identifies log entries with status codes that signify server errors. By using this filter, you can trigger a Lambda function to perform corrective actions, like restarting the instance or sending notifications.

Expected Output

If your site is accessible:

If your site is not accessible:

That’s all! By setting up a self-healing mechanism with AWS Lambda, EventBridge, and optionally CloudWatch, you can automate the detection and resolution of downtime issues for your website. This will help keep your website available and dependable for visitors. Happy coding!

* This newsletter was sourced from this Tutorials Dojo article.

* For more learning resources, you may visit: Free PlayCloud Guided Labs, Free AWS Digital Courses, and Free Practice Exams.