Celery

Celery is an asynchronous task queue for Python that allows you to execute tasks in the background, outside the main application flow. It is widely used for handling long-running or resource-intensive tasks without blocking the main application.

🔹 Key Features of Celery

✅ Asynchronous Processing – Runs tasks in the background without delaying user requests. ✅ Distributed Execution – Can run tasks across multiple worker machines. ✅ Scalability – Supports parallel processing to handle large workloads. ✅ Scheduled Tasks – Supports periodic task execution with celery-beat. ✅ Retry Mechanism – Can automatically retry failed tasks.

🛠️ How Celery Works

Celery follows a producer-consumer model with three main components:

1. Task Producer (Your Application) Defines and sends tasks to the message broker.
2. Message Broker (Redis / RabbitMQ / SQS, etc.) Queues and distributes tasks to workers.
3. Worker (Celery Workers) Listens to the broker and executes tasks asynchronously.

⚙️ Basic Setup in Django

1️⃣ Install Celery and a message broker (Redis)

pip install celery redis
        

2️⃣ Create a Celery instance (in project/celery.py)

from celery import Celery

app = Celery('project_name', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    return x + y
        

3️⃣ Run the Celery worker

celery -A project_name worker --loglevel=info
        

4️⃣ Call the task asynchronously

add.delay(10, 20)  # Executes in the background
        

🚀 When to Use Celery?

• Sending emails
• Processing large datasets
• Generating reports
• Scheduling periodic tasks (e.g., daily data backups)
• Web scraping