Part 2: Orchestrating the Pipeline – Airflow Integration & Anomaly Flagging

In Part 1, we built the foundation of a data quality monitoring system using Great Expectations (GE) integrated with PySpark. We defined validation rules and applied them to incoming raw data to detect anomalies and ensure only clean, trustworthy records reached downstream systems.

Now, it’s time to orchestrate this validation process using Apache Airflow a powerful workflow orchestration tool that allows us to schedule, monitor, and manage complex pipelines with ease.

In this part, we will walk through building an automated data pipeline that:

• Ingests raw data

• Executes Spark jobs integrated with GE

• Evaluates validation results

• Takes conditional action:

Let’s dive in.

What We'll Cover

Step 1: Setting up Apache Airflow

If you’re working locally, follow these steps to install and configure Airflow in a virtual environment:

Navigate to http://localhost:8080 and log in with your admin credentials.

Airflow uses Directed Acyclic Graphs (DAGs) to define workflows. Each DAG is composed of tasks, which can be Python scripts, Bash commands, Spark jobs, or API calls.

Step 2: Designing the Pipeline Logic

We want to build a DAG that handles the following:

1. Ingests raw data using Spark

2. Validates the data using GE expectations suite

3. Branches based on validation results:

Let’s sketch the structure:

Code Walkthrough – Building the DAG

Here’s the fully functional DAG broken into modular sections for better readability and scalability.

spark_validation_pipeline.py

Supporting Code: Modular Task Scripts

Inside your my_project/spark_jobs/ directory:

ingest_data.py

run_validation_with_ge.py

write_clean_data.py

store_bad_records.py

log_anomalies.py

Deliverables

By the end of this section, we’ve accomplished:

Task Status Built a DAG to orchestrate the full pipeline ✅ Integrated Spark with Great Expectations validation ✅ Implemented conditional branching in Airflow ✅ Stored bad records in quarantine zone ✅ Logged anomalies for downstream analysis ✅

📌 Final Thoughts

This approach ensures data quality gates are automated before any data reaches the final destination. Airflow adds modularity, observability, and recovery capabilities to the pipeline. You can now monitor task status, retry failures, and even trigger alerts, which we’ll explore next.

🔮 Coming Next: Part 3 – Self-Healing Pipelines

In the final part of this series, we’ll explore:

• Slack & email alerts for failed validations

• Auto-retries and backoff strategies

• Circuit breaker patterns and pipeline rollback logic

• Tracking validation metrics over time

🔔 Stay tuned for: 📘 Part 3: Self-Healing Pipelines – Slack Alerts, Retries & Recovery Logic