Deploying Airflow

What it does

This skill automates the deployment of Airflow DAGs and projects, streamlining workflows and ensuring consistent environments for your data pipelines. It supports both Astronomer's managed platform (Astro) and open-source Airflow deployments using Docker Compose or Kubernetes. The skill provides commands to deploy entire projects, just DAG files, only the Docker image, or dbt projects alongside Airflow.

When to use it

• When you need to quickly iterate on DAG file changes without rebuilding the entire Docker image.
• For production releases requiring a full project deployment including dependencies and configurations.
• In multi-repository CI/CD setups where images and DAGs are deployed independently.
• To deploy dbt projects alongside Airflow for data transformation workflows within Cosmos (on Astro).

Key capabilities

• Astro Deployment Commands: astro deploy, astro deploy --dags, astro deploy --image, astro deploy --dbt.
• DAG-Only Deployments: Fast deployments that only push DAG files.
• Image-Only Deployments: Pushes the Docker image without updating DAGs.
• GitHub Integration: Automated deployments triggered by branch merges (e.g., main -> production).
• Deploy Queue: Manages sequential deployment processing when multiple deploys are triggered.
• Open-Source Deployment Support: Provides guidance for deploying Airflow using Docker Compose and Kubernetes.

Example prompts

• "Deploy the current project to Astro."
• "Push only the DAG files without rebuilding the image."
• "Deploy the dbt project alongside Airflow on Astro."

Tips & gotchas

• For managed operations and faster CI/CD, use Astronomer's Astro platform.
• Use astro deploy --dags when you’ve only changed DAG files to avoid rebuilding the Docker image.
• Docker Compose is suitable for learning and exploration; Kubernetes with a Helm chart is recommended for production open-source Airflow deployments.