---
name: migrate-pyspark-to-snowpark-connect
description: |
  Migrate PySpark and Databricks workloads to Snowflake SCOS (Snowpark Connect for Spark).
  Use when: converting Spark code to run on Snowflake, analyzing PySpark compatibility,
  updating imports to Spark Connect equivalents, or migrating from Databricks.
  Generates SCOS-compatible reports (Issues.csv, InputFilesInventory.csv, ArtifactDependencyInventory.csv)
  for the dvp-scos-dashboard-generator using official SCOS EWI codes (SPRKCNTPY*).
  Triggers: migrate pyspark, convert spark, scos migration,
  spark connect, pyspark compatibility, snowpark connect.
parent_skill: snowpark-connect
allowed-tools: Read, Write, Bash
---

# Migrate PySpark to SCOS — Coordinator

Orchestrate a multi-phase migration of PySpark workloads to Snowflake SCOS (Snowpark Connect for Spark). This coordinator delegates work to specialist sub-agents and validates each phase with critic agents before advancing.

## When to Load

[snowpark-connect] Intent Detection: After user indicates migration intent (convert, migrate, update imports, rewrite for SCOS).

## Arguments

- `$ARGUMENTS` — Path to the PySpark file or directory to migrate

### Optional Metadata (from orchestrator)

| Parameter | Variable | Description |
|-----------|----------|-------------|
| Output path | `$OUTPUT` | Target directory for migrated files and Reports/ |
| Customer Email | `$EMAIL` | Project metadata for reports |
| Customer Company | `$COMPANY` | Project metadata for reports |
| Project Name | `$PROJECT` | Project name for reports |

If not provided, use `${ARGUMENTS}_scos` as output and prompt for metadata in Phase 4.

## Prerequisites

### Skill Directory

`<SKILL_DIRECTORY>` is the **parent** `snowpark-connect/` directory containing `pyproject.toml` and `scripts/`. All tool invocations use `uv run --project <SKILL_DIRECTORY>`.

### uv Package Manager

```bash
uv --version || curl -LsSf https://astral.sh/uv/install.sh | sh
```

## Workflow

You are a coordinator. You **NEVER** apply code fixes, update imports, or generate reports yourself. You delegate each phase to a specialist sub-agent via the `task()` tool, then validate with a critic sub-agent. State is tracked in `migration_state.json`.

### Phase 0: Collect Info and Create Conversion Folder

1. **Collect project info** from the user if not already provided: input path, output path, email, company, project name.

2. **Create timestamped conversion folder**:
```bash
OUTPUT_ROOT="<$OUTPUT or ${ARGUMENTS}_scos>"
TIMESTAMP=$(date +"%m-%d-%YT%H %M %S")
CONVERSION="${OUTPUT_ROOT}/Conversion-SCOS-${TIMESTAMP}"
mkdir -p "${CONVERSION}/Output" "${CONVERSION}/Reports" "${CONVERSION}/Logs"
```

3. **Copy source files** into `<CONVERSION>/Output/` (the `<MIGRATED>` directory):
```bash
cp -r $ARGUMENTS/* <CONVERSION>/Output/   # directory
cp $ARGUMENTS <CONVERSION>/Output/        # single file
```

4. **Build the file manifest**:
```bash
find <CONVERSION>/Output -name "*.py" -type f | sort
```

5. **Initialize git**:
```bash
cd <CONVERSION> && git init && git add . && git commit -m "Initial commit: source copied for SCOS migration" && git branch -M main
```

6. **Write `migration_state.json`** to `<CONVERSION>/`:
```json
{
  "phase": 0,
  "manifest": ["<list of .py files>"],
  "conversion_root": "<CONVERSION>",
  "migrated_dir": "<CONVERSION>/Output/",
  "skill_directory": "<SKILL_DIRECTORY>",
  "phases_completed": {},
  "metadata": {"email": "...", "company": "...", "project": "..."}
}
```

### Phase 1: Analysis

**Spawn specialist**: `task()` with the content of `agents/analyzer.md` as prompt context. Pass `migration_state.json` path. The analyzer runs `analyze_pyspark.py` and produces `analysis.json`.

**Spawn critic**: `task()` with `agents/analyzer-critic.md`. The critic validates `analysis.json` completeness and checks for known analyzer blind spots (UDFs, checkpoint, map subscript).

**Gate**: Critic must report `PASS`. If `FAIL`, re-run analyzer with critic feedback (max 2 retries). Update `migration_state.json` phase to 1.

**Git checkpoint**: `cd <CONVERSION> && git add -A && git commit -m "Phase 1: analysis complete"`

### Phase 2: Apply Fixes

**Spawn specialist**: `task()` with `agents/fixer.md`. The fixer reads `analysis.json`, loads `references/fix-rules.md` for the detailed rules, and applies fixes file-by-file. It updates `migration_state.json` with per-file progress.

**Spawn critic**: `task()` with `agents/fixer-critic.md`. The critic runs syntax checks, verifies all high-risk issues were addressed, and confirms no-ops were not over-annotated.

**Gate**: Critic must report `PASS` with zero syntax errors. If `FAIL`, re-run fixer on failing files with critic feedback. Update `migration_state.json` phase to 2.

**Git checkpoint**: `cd <CONVERSION> && git add -A && git commit -m "Phase 2: fixes applied"`

### Phase 3: Imports and Headers

**Spawn specialist**: `task()` with `agents/import-updater.md`. Updates imports, replaces session initialization with `snowpark_connect.init_spark_session()`, and adds migration header docstrings to every file.

**Spawn critic**: `task()` with `agents/import-critic.md`. Verifies every file has a migration header, no `SparkSession.builder` in non-comment code, entry point has `snowpark_connect`, no unsupported imports remain.

**Gate**: Critic must report `PASS`. Update `migration_state.json` phase to 3.

**Git checkpoint**: `cd <CONVERSION> && git add -A && git commit -m "Phase 3: imports and headers updated"`

### Phase 4: Generate Reports

**Spawn specialist**: `task()` with `agents/reporter.md`. Runs `generate_scos_reports.py` to produce `Reports/Issues.csv`, `Reports/InputFilesInventory.csv`, `Reports/ArtifactDependencyInventory.csv`.

**Spawn critic**: `task()` with `agents/reporter-critic.md`. Validates CSV files exist and have correct structure.

**Gate**: Critic must report `PASS`. Update `migration_state.json` phase to 4.

**Git checkpoint**: `cd <CONVERSION> && git add -A && git commit -m "Phase 4: reports generated"`

### Phase 5: Offer Validation (Optional)

Ask the user:
```
Migration complete! Would you like to validate the migrated workload
by running it end-to-end with synthetic data?
```

If yes, load `validate-pyspark-to-snowpark-connect/SKILL.md` with `<MIGRATED>` as `$ARGUMENTS`.

### Resumption

If context is lost mid-migration, read `migration_state.json` to determine the last completed phase and resume from the next one. The gate file contains the manifest, paths, and per-file progress needed to continue.

## Stopping Points

- Phase 0: After collecting project info — confirm settings before starting
- Phase 2: If critic fails after 2 retries — escalate to user with specific errors
- Phase 5: After migration completes — ask user about validation

## Success Criteria

- `migration_state.json` shows all phases 1-4 completed with critic approval
- `Reports/Issues.csv` exists with data rows
- `Reports/InputFilesInventory.csv` row count matches manifest
- All `.py` files pass `py_compile` syntax check
- Every `.py` file has a migration header docstring
- File count matches between original and migrated directories

## Output

```
<output_root>/
  Conversion-SCOS-<timestamp>/    ← <CONVERSION>
    Output/                       ← <MIGRATED> — converted files
    Reports/                      ← SCOS-compatible dashboard reports
    Logs/                         ← Migration log
    migration_state.json          ← Phase gate tracking
    analysis.json                 ← Compatibility analysis
```
