---
name: document-pipeline
description: "Set up automated document processing pipelines using Snowflake streams and tasks. Loaded by workflow skills (extraction, parsing, visual-analysis, classification) when user wants continuous processing."
parent_skill: document-intelligence
---

# Pipeline Sub-Skill

Set up automated document processing pipelines using Snowflake streams and tasks.

## When to Load

Load this reference when user wants to:
- Set up continuous document processing
- Automate extraction/parsing for new files
- Build production pipelines

---

## Template Routing

Each workflow skill (extraction.md, parsing.md, etc.) handles its own "Store Results" SQL. This file is loaded only when the user wants a **continuous processing pipeline**.

| Workflow just completed | Use Template |
|-------------------------|-------------|
| AI_EXTRACT (no page optimization) | Template A |
| AI_EXTRACT (with page filtering) | Template A2 |
| AI_PARSE_DOCUMENT | Template B |
| Visual Analysis (AI_COMPLETE) | Template C |
| AI_CLASSIFY | Template D |
| Multi-step chain (e.g., classify → extract) | Template E |

---

## Pipeline Setup

### Step 1: Page Range Configuration [WAIT]

**For AI_PARSE_DOCUMENT pipelines, ask:**

```
Do you want to parse the entire document or specific pages?

a. Entire document (all pages)
b. First page only
c. Specific page range (e.g., pages 1-10)
```

Wait for user response before proceeding.

**For AI_EXTRACT pipelines, ask:**

```
Do you want to extract from the entire document or specific pages?

a. Entire document (all pages)
b. First page only (common for invoices/forms)
c. Specific page range (e.g., pages 1-5)
```

Wait for user response before proceeding.

### Step 2: Pipeline Configuration [WAIT]

**Ask user:**

```
Configure your pipeline:
1. Warehouse name: [e.g., COMPUTE_WH]
2. Schedule frequency:
   - Every 1 minute
   - Every 5 minutes (recommended)
   - Every 15 minutes
   - Every hour
```

---

## Pipeline Templates

### Template A: AI_EXTRACT Pipeline

```sql
-- 1. Create results table
CREATE TABLE IF NOT EXISTS db.schema.extraction_results (
  result_id INT AUTOINCREMENT,
  file_path STRING,
  file_name STRING,
  extracted_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP(),
  field1 STRING,
  field2 STRING,
  raw_response VARIANT
);

-- 2. Create stream on stage
CREATE OR REPLACE STREAM db.schema.doc_stream 
  ON STAGE db.schema.my_stage;

-- 3. Create processing task
CREATE OR REPLACE TASK db.schema.extract_task
  WAREHOUSE = MY_WAREHOUSE
  SCHEDULE = '5 MINUTE'
  WHEN SYSTEM$STREAM_HAS_DATA('db.schema.doc_stream')
AS
  INSERT INTO db.schema.extraction_results (file_path, file_name, field1, field2, raw_response)
  SELECT 
    relative_path,
    SPLIT_PART(relative_path, '/', -1),
    result:field1::STRING,
    result:field2::STRING,
    result
  FROM db.schema.doc_stream,
  LATERAL (
    SELECT AI_EXTRACT(
      file => TO_FILE('@db.schema.my_stage', relative_path),
      responseFormat => {'field1': 'description', 'field2': 'description'}
    ) AS result
  )
  WHERE METADATA$ACTION = 'INSERT'
    AND relative_path LIKE '%.pdf';

-- 4. Resume task
ALTER TASK db.schema.extract_task RESUME;
```

### Template A2: AI_EXTRACT Pipeline with Page Optimization

Use this template when the user selected "First page only" or "Specific page range" in Step 1. It extracts only the relevant pages from each PDF before running AI_EXTRACT, reducing cost and improving performance. The `AFTER` clause ensures the extract task only runs after page extraction completes.

**Step 1: Create a stage for extracted pages**

```sql
CREATE STAGE IF NOT EXISTS db.schema.extracted_pages_stage
  DIRECTORY = (ENABLE = TRUE)
  ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE')
  COMMENT = 'Stage for extracted PDF pages';
```

**Step 2: Create the page extraction stored procedure**

```sql
CREATE OR REPLACE PROCEDURE db.schema.extract_pdf_pages(
    stage_name STRING, 
    file_name STRING, 
    dest_stage_name STRING,
    page_selection VARIANT  -- Can be: integer (single page), array of integers, or object with 'start' and 'end'
)
RETURNS VARIANT
LANGUAGE PYTHON
RUNTIME_VERSION = '3.9'
PACKAGES = ('snowflake-snowpark-python', 'pypdf2')
HANDLER = 'run'
AS
$
from PyPDF2 import PdfReader, PdfWriter
from snowflake.snowpark import Session
from snowflake.snowpark.file_operation import FileOperation
from io import BytesIO
import os
import json

def run(session: Session, stage_name: str, file_name: str, dest_stage_name: str, page_selection) -> dict:
    result = {
        "status": "success",
        "source_file": file_name,
        "total_pages": 0,
        "extracted_pages": [],
        "output_file": ""
    }
    
    try:
        # Download PDF from stage
        file_url = f"{stage_name}/{file_name}"
        get_result = session.file.get(file_url, '/tmp/')
        file_path = os.path.join('/tmp/', file_name)
        
        if not os.path.exists(file_path):
            return {"status": "error", "message": f"File {file_name} not found in {stage_name}"}

        with open(file_path, 'rb') as f:
            pdf_data = f.read()
            pdf_reader = PdfReader(BytesIO(pdf_data))
            num_pages = len(pdf_reader.pages)
            result["total_pages"] = num_pages
            
            # Determine which pages to extract (convert to 0-indexed)
            pages_to_extract = []
            
            if isinstance(page_selection, int):
                # Single page (1-indexed input)
                pages_to_extract = [page_selection - 1]
            elif isinstance(page_selection, list):
                # List of specific pages (1-indexed input)
                pages_to_extract = [p - 1 for p in page_selection]
            elif isinstance(page_selection, dict):
                # Range with 'start' and 'end' (1-indexed input)
                start = page_selection.get('start', 1) - 1
                end = min(page_selection.get('end', num_pages), num_pages)
                pages_to_extract = list(range(start, end))
            
            # Validate pages
            pages_to_extract = [p for p in pages_to_extract if 0 <= p < num_pages]
            
            if not pages_to_extract:
                return {"status": "error", "message": "No valid pages to extract"}
            
            # Create new PDF with selected pages
            writer = PdfWriter()
            for page_num in pages_to_extract:
                writer.add_page(pdf_reader.pages[page_num])
                result["extracted_pages"].append(page_num + 1)  # Return 1-indexed
            
            # Generate output filename
            base_name = os.path.splitext(file_name)[0]
            if len(pages_to_extract) == 1:
                output_filename = f'{base_name}_page_{pages_to_extract[0] + 1}.pdf'
            else:
                output_filename = f'{base_name}_pages_{pages_to_extract[0] + 1}_to_{pages_to_extract[-1] + 1}.pdf'
            
            output_path = os.path.join('/tmp/', output_filename)
            
            with open(output_path, 'wb') as output_file:
                writer.write(output_file)
            
            # Upload to destination stage
            FileOperation(session).put(
                f"file://{output_path}",
                dest_stage_name,
                auto_compress=False
            )
            
            result["output_file"] = output_filename
            
            # Cleanup
            os.remove(file_path)
            os.remove(output_path)
                
        return result
        
    except Exception as e:
        return {"status": "error", "message": str(e)}
$;
```

**Page selection examples** — use these values for the `page_selection` parameter in the task below:

```sql
-- First page only
CALL db.schema.extract_pdf_pages(
  '@db.schema.my_stage',
  'invoice.pdf',
  '@db.schema.extracted_pages_stage',
  1  -- Single page (1-indexed)
);

-- Page range (e.g., pages 1-5)
CALL db.schema.extract_pdf_pages(
  '@db.schema.my_stage',
  'invoice.pdf',
  '@db.schema.extracted_pages_stage',
  {'start': 1, 'end': 5}  -- Range (1-indexed, inclusive)
);

-- Specific pages (e.g., pages 1, 3, 10)
CALL db.schema.extract_pdf_pages(
  '@db.schema.my_stage',
  'invoice.pdf',
  '@db.schema.extracted_pages_stage',
  [1, 3, 10]  -- Array of pages (1-indexed)
);
```

**Step 3: Create the pipeline**

```sql
-- 1. Create results table
CREATE TABLE IF NOT EXISTS db.schema.extraction_results (
  result_id INT AUTOINCREMENT,
  file_path STRING,
  file_name STRING,
  extracted_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP(),
  field1 STRING,
  field2 STRING,
  raw_response VARIANT
);

-- 2. Create stream on source stage (watches for new uploads)
CREATE OR REPLACE STREAM db.schema.doc_stream 
  ON STAGE db.schema.my_stage;

-- 3. Create stream on extracted pages stage (watches for extracted PDFs)
CREATE OR REPLACE STREAM db.schema.extracted_pages_stream 
  ON STAGE db.schema.extracted_pages_stage;

-- 4. Create page extraction task (runs stored procedure per file)
--    Replace page_selection with user's choice: 1 for first page, {'start': 1, 'end': 5} for range
CREATE OR REPLACE TASK db.schema.page_extract_task
  WAREHOUSE = MY_WAREHOUSE
  SCHEDULE = '5 MINUTE'
  WHEN SYSTEM$STREAM_HAS_DATA('db.schema.doc_stream')
AS
  DECLARE
    c CURSOR FOR 
      SELECT relative_path 
      FROM db.schema.doc_stream 
      WHERE METADATA$ACTION = 'INSERT' 
        AND relative_path LIKE '%.pdf';
    file_name STRING;
  BEGIN
    FOR record IN c DO
      BEGIN
        file_name := SPLIT_PART(record.relative_path, '/', -1);
        CALL db.schema.extract_pdf_pages(
          '@db.schema.my_stage',
          :file_name,
          '@db.schema.extracted_pages_stage',
          1  -- Replace with user's page selection
        );
      EXCEPTION
        WHEN OTHER THEN
          -- Log error and continue processing remaining files
          LET err_msg := SQLERRM;
      END;
    END FOR;
    -- Refresh so extracted files are visible to the next task
    ALTER STAGE db.schema.extracted_pages_stage REFRESH;
  END;

-- 5. Create AI_EXTRACT task (runs AFTER page extraction completes)
CREATE OR REPLACE TASK db.schema.extract_task
  WAREHOUSE = MY_WAREHOUSE
  AFTER db.schema.page_extract_task
AS
  INSERT INTO db.schema.extraction_results (file_path, file_name, field1, field2, raw_response)
  SELECT 
    relative_path,
    SPLIT_PART(relative_path, '/', -1),
    result:field1::STRING,
    result:field2::STRING,
    result
  FROM db.schema.extracted_pages_stream,
  LATERAL (
    SELECT AI_EXTRACT(
      file => TO_FILE('@db.schema.extracted_pages_stage', relative_path),
      responseFormat => {'field1': 'description', 'field2': 'description'}
    ) AS result
  )
  WHERE METADATA$ACTION = 'INSERT'
    AND relative_path LIKE '%.pdf';

-- 6. Resume both tasks (resume child first, then parent)
ALTER TASK db.schema.extract_task RESUME;
ALTER TASK db.schema.page_extract_task RESUME;
```

### Template B: AI_PARSE_DOCUMENT Pipeline

```sql
-- 1. Create results table
CREATE TABLE IF NOT EXISTS db.schema.parsed_documents (
  doc_id INT AUTOINCREMENT,
  file_path STRING,
  file_name STRING,
  content TEXT,
  parsed_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

-- 2. Create stream on stage
CREATE OR REPLACE STREAM db.schema.parse_stream 
  ON STAGE db.schema.my_stage;

-- 3. Create processing task
CREATE OR REPLACE TASK db.schema.parse_task
  WAREHOUSE = MY_WAREHOUSE
  SCHEDULE = '5 MINUTE'
  WHEN SYSTEM$STREAM_HAS_DATA('db.schema.parse_stream')
AS
  INSERT INTO db.schema.parsed_documents (file_path, file_name, content)
  SELECT 
    relative_path,
    SPLIT_PART(relative_path, '/', -1),
    AI_PARSE_DOCUMENT(TO_FILE('@db.schema.my_stage', relative_path), {'mode': 'LAYOUT'}):content::STRING
  FROM db.schema.parse_stream
  WHERE METADATA$ACTION = 'INSERT'
    AND relative_path LIKE '%.pdf';

-- 4. Resume task
ALTER TASK db.schema.parse_task RESUME;
```

### Template C: Visual Analysis Pipeline

```sql
-- 1. Create results table
CREATE TABLE IF NOT EXISTS db.schema.visual_results (
  result_id INT AUTOINCREMENT,
  image_path STRING,
  analysis_result TEXT,
  analyzed_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

-- 2. Create stream on images stage
CREATE OR REPLACE STREAM db.schema.images_stream 
  ON STAGE db.schema.images_stage;

-- 3. Create processing task
CREATE OR REPLACE TASK db.schema.analyze_task
  WAREHOUSE = MY_WAREHOUSE
  SCHEDULE = '5 MINUTE'
  WHEN SYSTEM$STREAM_HAS_DATA('db.schema.images_stream')
AS
  INSERT INTO db.schema.visual_results (image_path, analysis_result)
  SELECT 
    relative_path,
    AI_COMPLETE('claude-3-5-sonnet', 'Analyze this image...', TO_FILE('@db.schema.images_stage', relative_path))
  FROM db.schema.images_stream
  WHERE METADATA$ACTION = 'INSERT'
    AND (relative_path LIKE '%.png' OR relative_path LIKE '%.jpg');

-- 4. Resume task
ALTER TASK db.schema.analyze_task RESUME;
```

### Template D: AI_CLASSIFY Pipeline

Classifies new documents as they arrive. Supports both image files (direct classification) and document files (parse first page, then classify text).

```sql
-- 1. Create results table
CREATE TABLE IF NOT EXISTS db.schema.classification_results (
  result_id INT AUTOINCREMENT,
  file_path STRING,
  file_name STRING,
  labels VARIANT,
  classified_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

-- 2. Create stream on stage
CREATE OR REPLACE STREAM db.schema.classify_stream 
  ON STAGE db.schema.my_stage;

-- 3. Create classification task (for document files — parse first page then classify)
CREATE OR REPLACE TASK db.schema.classify_task
  WAREHOUSE = MY_WAREHOUSE
  SCHEDULE = '5 MINUTE'
  WHEN SYSTEM$STREAM_HAS_DATA('db.schema.classify_stream')
AS
  INSERT INTO db.schema.classification_results (file_path, file_name, labels)
  SELECT 
    relative_path,
    SPLIT_PART(relative_path, '/', -1),
    AI_CLASSIFY(
      AI_PARSE_DOCUMENT(
        TO_FILE('@db.schema.my_stage', SPLIT_PART(relative_path, '/', -1)),
        {'mode': 'LAYOUT', 'page_filter': [{'start': 0, 'end': 1}]}
      ):pages[0]:content::VARCHAR,
      ['category1', 'category2', 'category3'],
      {'task_description': 'Classify documents by type'}
    ):labels
  FROM db.schema.classify_stream
  WHERE METADATA$ACTION = 'INSERT'
    AND relative_path LIKE '%.pdf';

-- 4. Resume task
ALTER TASK db.schema.classify_task RESUME;
```

**For image files**, replace the task body with direct classification (no parsing needed):

```sql
CREATE OR REPLACE TASK db.schema.classify_task
  WAREHOUSE = MY_WAREHOUSE
  SCHEDULE = '5 MINUTE'
  WHEN SYSTEM$STREAM_HAS_DATA('db.schema.classify_stream')
AS
  INSERT INTO db.schema.classification_results (file_path, file_name, labels)
  SELECT 
    relative_path,
    SPLIT_PART(relative_path, '/', -1),
    AI_CLASSIFY(
      TO_FILE('@db.schema.my_stage', SPLIT_PART(relative_path, '/', -1)),
      ['category1', 'category2', 'category3']
    ):labels
  FROM db.schema.classify_stream
  WHERE METADATA$ACTION = 'INSERT'
    AND (relative_path LIKE '%.jpg' OR relative_path LIKE '%.png');

ALTER TASK db.schema.classify_task RESUME;
```

**For mixed file types**, create two tasks: one for documents (parse + classify) and one for images (direct classify), both reading from the same stream.

---

### Template E: Multi-Step Pipeline (Chained Operations)

Use when the user's session involved multiple chained operations (e.g., classify → extract, classify → extract → summarize). This creates an N-task pipeline where each task's output drives the next task via Snowflake's `AFTER` clause.

**When to detect:** If the user went through two or more sequential operations (e.g., classification then extraction, or classification then extraction then summarization), offer this template instead of individual templates.

**Architecture:**

```
New file uploaded to stage
        │
        ▼
┌─────────────────────────┐
│  Task 1 (root)          │  SCHEDULE + WHEN SYSTEM$STREAM_HAS_DATA
│  First operation         │  e.g., AI_CLASSIFY
│  → Results Table 1       │
└────────────┬────────────┘
             │ AFTER
             ▼
┌─────────────────────────┐
│  Task 2 (child)         │  AFTER task_1
│  Second operation        │  e.g., AI_EXTRACT
│  Filters by Task 1      │
│  → Results Table 2       │
└────────────┬────────────┘
             │ AFTER (optional — add more steps as needed)
             ▼
┌─────────────────────────┐
│  Task N (child)         │  AFTER task_(N-1)
│  Nth operation           │  e.g., AI_COMPLETE
│  Filters by Task N-1    │
│  → Results Table N       │
└─────────────────────────┘
```

**Key rules:**
- Only the **root task** (Task 1) has `SCHEDULE` and `WHEN SYSTEM$STREAM_HAS_DATA`
- All child tasks use `AFTER parent_task` — no schedule, no stream condition
- If the root task skips (stream empty), the entire chain is skipped
- If any task fails, downstream tasks do **not** run

**Example: Classify → Extract (most common)**

This pipeline classifies new PDFs, then extracts fields only from documents matching a specific category (e.g., "Annual report").

```sql
-- 1. Create classification results table
CREATE TABLE IF NOT EXISTS db.schema.classification_results (
  result_id INT AUTOINCREMENT,
  file_path STRING,
  file_name STRING,
  labels VARIANT,
  classified_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

-- 2. Create extraction results table
CREATE TABLE IF NOT EXISTS db.schema.extraction_results (
  result_id INT AUTOINCREMENT,
  file_path STRING,
  file_name STRING,
  extracted_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP(),
  -- Add user's extraction fields here
  field1 STRING,
  field2 STRING,
  raw_response VARIANT
);

-- 3. Create stream on source stage
CREATE OR REPLACE STREAM db.schema.doc_stream
  ON STAGE db.schema.my_stage;

-- 4. Create classification task (parent, scheduled)
CREATE OR REPLACE TASK db.schema.classify_task
  WAREHOUSE = MY_WAREHOUSE
  SCHEDULE = '5 MINUTE'
  WHEN SYSTEM$STREAM_HAS_DATA('db.schema.doc_stream')
AS
  INSERT INTO db.schema.classification_results (file_path, file_name, labels)
  SELECT 
    relative_path,
    SPLIT_PART(relative_path, '/', -1),
    AI_CLASSIFY(
      AI_PARSE_DOCUMENT(
        TO_FILE('@db.schema.my_stage', SPLIT_PART(relative_path, '/', -1)),
        {'mode': 'LAYOUT', 'page_filter': [{'start': 0, 'end': 1}]}
      ):pages[0]:content::VARCHAR,
      ['category1', 'category2'],
      {'task_description': 'Classify documents by type'}
    ):labels
  FROM db.schema.doc_stream
  WHERE METADATA$ACTION = 'INSERT'
    AND relative_path LIKE '%.pdf';

-- 5. Create extraction task (child, runs AFTER classification)
--    Only processes documents matching the target category
CREATE OR REPLACE TASK db.schema.extract_task
  WAREHOUSE = MY_WAREHOUSE
  AFTER db.schema.classify_task
AS
  INSERT INTO db.schema.extraction_results (file_path, file_name, field1, field2, raw_response)
  SELECT 
    c.file_path,
    c.file_name,
    result:response:field1::STRING,
    result:response:field2::STRING,
    result
  FROM db.schema.classification_results c,
  LATERAL (
    SELECT AI_EXTRACT(
      file => TO_FILE('@db.schema.my_stage', c.file_name),
      responseFormat => {
        'field1': 'Description of field 1',
        'field2': 'Description of field 2'
      }
    ) AS result
  )
  WHERE c.labels[0]::VARCHAR = 'target_category'
    AND c.classified_at >= DATEADD('day', -1, CURRENT_TIMESTAMP());

-- 6. Resume tasks (innermost child first → parent)
ALTER TASK db.schema.extract_task RESUME;
ALTER TASK db.schema.classify_task RESUME;
```

**Example: Classify → Extract → Summarize (3-step)**

This pipeline classifies new PDFs, extracts fields from a target category, then summarizes the extracted data.

```sql
-- 1. Create tables (classification_results and extraction_results same as above)

-- 2. Create summary results table
CREATE TABLE IF NOT EXISTS db.schema.summary_results (
  result_id INT AUTOINCREMENT,
  file_path STRING,
  file_name STRING,
  summarized_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP(),
  summary STRING,
  raw_response VARIANT
);

-- 3. Create stream on source stage
CREATE OR REPLACE STREAM db.schema.doc_stream
  ON STAGE db.schema.my_stage;

-- 4. Create classification task (root, scheduled)
--    Same as 2-step example above

-- 5. Create extraction task (child, AFTER classify)
--    Same as 2-step example above

-- 6. Create summary task (grandchild, AFTER extract)
CREATE OR REPLACE TASK db.schema.summarize_task
  WAREHOUSE = MY_WAREHOUSE
  AFTER db.schema.extract_task
AS
  INSERT INTO db.schema.summary_results (file_path, file_name, summary, raw_response)
  SELECT 
    e.file_path,
    e.file_name,
    ai_result
  FROM db.schema.extraction_results e,
  LATERAL (
    SELECT AI_COMPLETE(
      'claude-3-5-sonnet',
      'Summarize the following extracted data in 2-3 sentences: ' ||
      OBJECT_CONSTRUCT('field1', e.field1, 'field2', e.field2)::STRING
    ) AS ai_result
  )
  WHERE e.extracted_at >= DATEADD('day', -1, CURRENT_TIMESTAMP());

-- 7. Resume tasks (innermost child first → parent)
ALTER TASK db.schema.summarize_task RESUME;
ALTER TASK db.schema.extract_task RESUME;
ALTER TASK db.schema.classify_task RESUME;
```

**Pattern for adding more steps:** Each additional step follows the same pattern — create a results table, create a task with `AFTER previous_task`, and filter by the previous table's timestamp.

**Adapting for other combinations:**

| Steps | Combination | Task 1 (root) | Task 2 | Task 3 | Filter |
|-------|-------------|----------------|--------|--------|--------|
| 2 | Classify → Extract | AI_CLASSIFY | AI_EXTRACT | — | `category = 'target'` |
| 2 | Classify → Parse | AI_CLASSIFY | AI_PARSE_DOCUMENT | — | `category = 'target'` |
| 2 | Classify → Visual | AI_CLASSIFY | AI_COMPLETE | — | `category = 'target'` |
| 3 | Classify → Extract → Summarize | AI_CLASSIFY | AI_EXTRACT | AI_COMPLETE | Each filters by previous |
| N | Any chain | Step 1 | Step 2 | ... Step N | Each task uses `AFTER` on its predecessor and filters the previous results table |

To adapt: replace each task body with the appropriate AI function call, keeping the `AFTER` chaining. Each child task queries the previous step's results table with a recency filter (`WHERE timestamp >= DATEADD('day', -1, CURRENT_TIMESTAMP())`).

**Management (suspend root first, resume innermost child first):**

```sql
-- Suspend (root first → children in order)
ALTER TASK db.schema.classify_task SUSPEND;
ALTER TASK db.schema.extract_task SUSPEND;
ALTER TASK db.schema.summarize_task SUSPEND;  -- add more as needed

-- Resume (innermost child first → root last)
ALTER TASK db.schema.summarize_task RESUME;   -- add more as needed
ALTER TASK db.schema.extract_task RESUME;
ALTER TASK db.schema.classify_task RESUME;
```

**Rule:** Always suspend from root → leaf. Always resume from leaf → root. This prevents a parent from completing before its children are active.

---

## Pipeline Management

### Monitor Status

```sql
-- Check task history
SELECT * FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
  TASK_NAME => 'extract_task',
  SCHEDULED_TIME_RANGE_START => DATEADD('hour', -24, CURRENT_TIMESTAMP())
)) ORDER BY SCHEDULED_TIME DESC;

-- Check stream status
SHOW STREAMS LIKE '%stream%';

-- View pending files
SELECT * FROM db.schema.doc_stream;
```

### Pause/Resume

```sql
-- Template A (single task)
ALTER TASK db.schema.extract_task SUSPEND;  -- Pause
ALTER TASK db.schema.extract_task RESUME;   -- Resume

-- Template A2 (page optimization — manage both tasks)
-- Suspend parent first, then child
ALTER TASK db.schema.page_extract_task SUSPEND;
ALTER TASK db.schema.extract_task SUSPEND;
-- Resume child first, then parent
ALTER TASK db.schema.extract_task RESUME;
ALTER TASK db.schema.page_extract_task RESUME;
```

### Modify Schedule

```sql
-- Template A
ALTER TASK db.schema.extract_task SET SCHEDULE = '15 MINUTE';

-- Template A2 (only the parent task has a schedule)
ALTER TASK db.schema.page_extract_task SET SCHEDULE = '15 MINUTE';
```

---

## Troubleshooting

| Issue | Solution |
|-------|----------|
| Task not running | Check if resumed: `SHOW TASKS LIKE 'task_name';` |
| Stream shows no data | Refresh stage: `ALTER STAGE stage_name REFRESH;` |
| Extraction errors | Check task history for error messages |

---

## Stopping Points

| After Step | Wait For |
|------------|----------|
| Post-processing options | User choice (Done/Store/Pipeline) |
| Store results | User confirmation to set up pipeline |
| Page optimization (AI_EXTRACT only) | User response |
| Page range (AI_PARSE_DOCUMENT only) | User response |
| Pipeline configuration | Warehouse and schedule selection |

## Output

Automated pipeline with stream, task, and results table for continuous document processing.
