---
name: airflow
description: Manages Apache Airflow operations including listing, testing, running, and debugging DAGs, viewing task logs, checking connections and variables, and monitoring system health. Use when working with Airflow DAGs, pipelines, workflows, or tasks, or when the user mentions testing dags, running pipelines, debugging workflows, dag failures, task errors, dag status, pipeline status, list dags, show connections, check variables, or airflow health.
---

# Airflow Operations

Use `cortex airflow` commands to query, manage, and troubleshoot Airflow workflows.

## Running the CLI

```bash
cortex airflow <command>
```

## Instance Configuration

Manage multiple Airflow instances with persistent configuration:

```bash
# Add a new instance
cortex airflow instance add prod --url https://airflow.example.com --token "$API_TOKEN"
cortex airflow instance add staging --url https://staging.example.com --username admin --password admin

# List and switch instances
cortex airflow instance list      # Shows all instances in a table
cortex airflow instance use prod  # Switch to prod instance
cortex airflow instance current   # Show current instance
cortex airflow instance delete old-instance

# Auto-discover instances (use --dry-run to preview first)
cortex airflow instance discover --dry-run        # Preview all discoverable instances
cortex airflow instance discover                  # Discover from all backends (astro, local)
cortex airflow instance discover astro            # Discover Astro deployments only
cortex airflow instance discover astro --all-workspaces  # Include all accessible workspaces
cortex airflow instance discover local            # Scan common local Airflow ports
cortex airflow instance discover local --scan     # Deep scan all ports 1024-65535

# IMPORTANT: Always run with --dry-run first and ask for user consent before
# running discover without it. The non-dry-run mode creates API tokens in
# Astro Cloud, which is a sensitive action that requires explicit approval.

# Override instance for a single command
cortex airflow --instance staging dags list
```

Config file: `~/.af/config.yaml` (override with `--config` or `AF_CONFIG` env var)

Tokens in config can reference environment variables using `${VAR}` syntax:
```yaml
instances:
- name: prod
  url: https://airflow.example.com
  auth:
    token: ${AIRFLOW_API_TOKEN}
```

Or use environment variables directly (no config file needed):

```bash
export AIRFLOW_API_URL=http://localhost:8080
export AIRFLOW_AUTH_TOKEN=your-token-here
# Or username/password:
export AIRFLOW_USERNAME=admin
export AIRFLOW_PASSWORD=admin
```

Or CLI flags: `cortex airflow --airflow-url http://localhost:8080 --token "$TOKEN" <command>`

## Quick Reference

| Command | Description |
|---------|-------------|
| `cortex airflow health` | System health check |
| `cortex airflow dags list` | List all DAGs |
| `cortex airflow dags get <dag_id>` | Get DAG details |
| `cortex airflow dags explore <dag_id>` | Full DAG investigation |
| `cortex airflow dags source <dag_id>` | Get DAG source code |
| `cortex airflow dags pause <dag_id>` | Pause DAG scheduling |
| `cortex airflow dags unpause <dag_id>` | Resume DAG scheduling |
| `cortex airflow dags errors` | List import errors |
| `cortex airflow dags warnings` | List DAG warnings |
| `cortex airflow dags stats` | DAG run statistics |
| `cortex airflow runs list` | List DAG runs |
| `cortex airflow runs get <dag_id> <run_id>` | Get run details |
| `cortex airflow runs trigger <dag_id>` | Trigger a DAG run |
| `cortex airflow runs trigger-wait <dag_id>` | Trigger and wait for completion |
| `cortex airflow runs diagnose <dag_id> <run_id>` | Diagnose failed run |
| `cortex airflow tasks list <dag_id>` | List tasks in DAG |
| `cortex airflow tasks get <dag_id> <task_id>` | Get task definition |
| `cortex airflow tasks instance <dag_id> <run_id> <task_id>` | Get task instance |
| `cortex airflow tasks logs <dag_id> <run_id> <task_id>` | Get task logs |
| `cortex airflow config version` | Airflow version |
| `cortex airflow config show` | Full configuration |
| `cortex airflow config connections` | List connections |
| `cortex airflow config variables` | List variables |
| `cortex airflow config variable <key>` | Get specific variable |
| `cortex airflow config pools` | List pools |
| `cortex airflow config pool <name>` | Get pool details |
| `cortex airflow config plugins` | List plugins |
| `cortex airflow config providers` | List providers |
| `cortex airflow config assets` | List assets/datasets |
| `cortex airflow api <endpoint>` | Direct REST API access |
| `cortex airflow api ls` | List available API endpoints |
| `cortex airflow api ls --filter X` | List endpoints matching pattern |

## User Intent Patterns

### DAG Operations
- "What DAGs exist?" / "List all DAGs" -> `cortex airflow dags list`
- "Tell me about DAG X" / "What is DAG Y?" -> `cortex airflow dags explore <dag_id>`
- "What's the schedule for DAG X?" -> `cortex airflow dags get <dag_id>`
- "Show me the code for DAG X" -> `cortex airflow dags source <dag_id>`
- "Stop DAG X" / "Pause this workflow" -> `cortex airflow dags pause <dag_id>`
- "Resume DAG X" -> `cortex airflow dags unpause <dag_id>`
- "Are there any DAG errors?" -> `cortex airflow dags errors`

### Run Operations
- "What runs have executed?" -> `cortex airflow runs list`
- "Run DAG X" / "Trigger the pipeline" -> `cortex airflow runs trigger <dag_id>`
- "Run DAG X and wait" -> `cortex airflow runs trigger-wait <dag_id>`
- "Why did this run fail?" -> `cortex airflow runs diagnose <dag_id> <run_id>`

### Task Operations
- "What tasks are in DAG X?" -> `cortex airflow tasks list <dag_id>`
- "Get task logs" / "Why did task fail?" -> `cortex airflow tasks logs <dag_id> <run_id> <task_id>`

### System Operations
- "What version of Airflow?" -> `cortex airflow config version`
- "What connections exist?" -> `cortex airflow config connections`
- "Are pools full?" -> `cortex airflow config pools`
- "Is Airflow healthy?" -> `cortex airflow health`

### API Exploration
- "What API endpoints are available?" -> `cortex airflow api ls`
- "Find variable endpoints" -> `cortex airflow api ls --filter variable`
- "Access XCom values" / "Get XCom" -> `cortex airflow api xcom-entries -F dag_id=X -F task_id=Y`
- "Get event logs" / "Audit trail" -> `cortex airflow api event-logs -F dag_id=X`
- "Create connection via API" -> `cortex airflow api connections -X POST --body '{...}'`
- "Create variable via API" -> `cortex airflow api variables -X POST -F key=name -f value=val`

## Common Workflows

### Investigate a Failed Run

```bash
# 1. List recent runs to find failure
cortex airflow runs list --dag-id my_dag

# 2. Diagnose the specific run
cortex airflow runs diagnose my_dag manual__2024-01-15T10:00:00+00:00

# 3. Get logs for failed task (from diagnose output)
cortex airflow tasks logs my_dag manual__2024-01-15T10:00:00+00:00 extract_data
```

### Morning Health Check

```bash
# 1. Overall system health
cortex airflow health

# 2. Check for broken DAGs
cortex airflow dags errors

# 3. Check pool utilization
cortex airflow config pools
```

### Understand a DAG

```bash
# Get comprehensive overview (metadata + tasks + source)
cortex airflow dags explore my_dag
```

### Check Why DAG Isn't Running

```bash
# Check if paused
cortex airflow dags get my_dag

# Check for import errors
cortex airflow dags errors

# Check recent runs
cortex airflow runs list --dag-id my_dag
```

### Trigger and Monitor

```bash
# Option 1: Trigger and wait (blocking)
cortex airflow runs trigger-wait my_dag --timeout 1800

# Option 2: Trigger and check later
cortex airflow runs trigger my_dag
cortex airflow runs get my_dag <run_id>
```

## Output Format

All commands output JSON (except `instance` commands which use human-readable tables):

```bash
cortex airflow dags list
# {
#   "total_dags": 5,
#   "returned_count": 5,
#   "dags": [...]
# }
```

Use `jq` for filtering:

```bash
# Find failed runs
cortex airflow runs list | jq '.dag_runs[] | select(.state == "failed")'

# Get DAG IDs only
cortex airflow dags list | jq '.dags[].dag_id'

# Find paused DAGs
cortex airflow dags list | jq '[.dags[] | select(.is_paused == true)]'
```

## Task Logs Options

```bash
# Get logs for specific retry attempt
cortex airflow tasks logs my_dag run_id task_id --try 2

# Get logs for mapped task index
cortex airflow tasks logs my_dag run_id task_id --map-index 5
```

## Direct API Access with `cortex airflow api`

Use `cortex airflow api` for endpoints not covered by high-level commands (XCom, event-logs, backfills, etc).

```bash
# Discover available endpoints
cortex airflow api ls
cortex airflow api ls --filter variable

# Basic usage
cortex airflow api dags
cortex airflow api dags -F limit=10 -F only_active=true
cortex airflow api variables -X POST -F key=my_var -f value="my value"
cortex airflow api variables/old_var -X DELETE
```

**Field syntax**: `-F key=value` auto-converts types, `-f key=value` keeps as string.

**Full reference**: See [api-reference.md](api-reference.md) for all options, common endpoints (XCom, event-logs, backfills), and examples.

## Related Skills

- `testing-dags` - Test DAGs with debugging and fixing cycles
- `debugging-dags` - Comprehensive DAG failure diagnosis and root cause analysis
- `authoring-dags` - Creating and editing DAG files with best practices
- `managing-astro-local-env` - Starting/stopping local Airflow environment
