# Extraction Workflow

Structured field and table extraction using AI_EXTRACT.

## Use When

- User wants specific fields (names, dates, amounts, IDs)
- User wants tables with defined columns (line items, transactions)

## Constraints & Pricing

See [AI_EXTRACT docs](https://docs.snowflake.com/en/sql-reference/functions/ai_extract) for constraints (max file size, pages, questions per call) and [AI Functions Costs](https://docs.snowflake.com/en/user-guide/snowflake-cortex/aisql#costs) for pricing details.

**Quick reference:** 5 credits per 1M tokens. With `scale_factor`, multiply cost by the factor value.

## Reference
Read `../../references/ai-extract.md` for TO_FILE path handling rules and batch processing patterns. See [AI_EXTRACT docs](https://docs.snowflake.com/en/sql-reference/functions/ai_extract) for full syntax and parameters.

---

## Workflow

### 1. Get File Location [WAIT]

**If files are on Snowflake stage:** Get full stage path, proceed to Step 2.

**If files are local:** You must get the upload destination from the user. Do not create any stages or run any SQL until the user provides this information.

Ask: "Which database, schema, and stage name should I use? (e.g., MY_DB.MY_SCHEMA.MY_STAGE)"

Use the exact stage name the user provides. After user responds, create stage with server-side encryption:
```sql
CREATE STAGE IF NOT EXISTS db.schema.user_provided_stage_name 
DIRECTORY = (ENABLE = TRUE) 
ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');
```

Then upload the files.

### 2. Define Extraction Fields [WAIT]

Ask what fields to extract. For each field, get:
- Field name
- Description of what to extract
- Type: single value, list, or table

Confirm fields back to user before proceeding.

### 3. List Files

Show available files in the stage. Inform user you'll test on one file first.

### 4. Cost Estimate

Display estimated cost for the test file, then proceed to test.

### 5. Single File Test [WAIT]

Extract from ONE file only. Display results clearly.
## ⚠️ CRITICAL: always use the AI_* function names WITHOUT the `SNOWFLAKE.CORTEX` namespace prefix

Ask if satisfied:
- **Yes** → proceed to Step 6 (batch)
- **No, wrong fields extracted** → return to Step 2 to refine field definitions
- **No, OCR quality issues** (typos, garbled characters, missing values that clearly exist in the document) → proceed to Step 5a (Scale Factor Tuning)

**How to detect OCR issues vs field definition issues:**
- OCR issues: values are garbled (e.g., "lnv0ice" instead of "Invoice"), characters are swapped, or fields return null even though the data visibly exists in the document
- Field issues: wrong field extracted, wrong format, or irrelevant data returned

### 5a. Scale Factor Tuning [WAIT]

When OCR quality is the problem, iteratively increase `scale_factor` to find the minimum effective value. The sequence to try is: **1.5 → 2.0 → 2.5 → 3.0 → 4.0**.

**Procedure:**

1. **Start at scale_factor 1.5.** Display the cost impact before running:
   ```
   Retrying with scale_factor: 1.5
   - Cost: 1.5x the default (~X credits for this file)
   - Max pages per call: 83 (down from 125)
   ```
   Re-run the same single-file extraction with `config => { 'scale_factor': 1.5 }`.

2. **Show results and ask user:**
   - **Good enough** → lock in this scale_factor value, proceed to Step 6
   - **Better but not there yet** → bump to next value in sequence, show new cost impact, re-run
   - **No improvement / worse** → the issue is likely not OCR-related. Suggest going back to Step 2 to refine field definitions, or trying AI_PARSE_DOCUMENT + AI_COMPLETE as an alternative approach

3. **Repeat** until user is satisfied or scale_factor 4.0 is reached.

4. **If 4.0 is reached and still not satisfactory:** Inform user this is the maximum scale_factor. Suggest:
   - Refining field descriptions (Step 2)
   - Using AI_PARSE_DOCUMENT (OCR mode) to get full text, then AI_COMPLETE to extract fields from that text
   - Checking if the source document quality can be improved

**Important:** Always display the cost impact before each scale_factor retry. Never silently increase scale_factor. See [AI_EXTRACT docs](https://docs.snowflake.com/en/sql-reference/functions/ai_extract) for the scale_factor cost/page tradeoff table.

### 6. Batch Process

If a scale_factor was determined in Step 5a, carry it into the batch query using the `config` parameter.

Display batch cost for all files. **If scale_factor > 1.0, show both the default cost and the scaled cost** so the user understands the tradeoff:

```
Batch cost estimate (X files):
- At default (scale_factor 1.0): ~Y credits
- At scale_factor Z: ~Y × Z credits
```

Then execute batch extraction (with `config => { 'scale_factor': <value> }` if applicable).

### 7. Post-Processing [WAIT]

Offer options:
1. Done - I have what I need
2. Store results in a Snowflake table
3. Set up a pipeline for continuous processing

**Option 2 — Store results:**

```sql
CREATE TABLE IF NOT EXISTS db.schema.extraction_results (
  result_id INT AUTOINCREMENT,
  file_name STRING,
  extracted_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP(),
  -- Add user's extraction fields here
  field1 STRING,
  field2 STRING,
  raw_response VARIANT
);

INSERT INTO db.schema.extraction_results (file_name, field1, field2, raw_response)
SELECT 
  SPLIT_PART(relative_path, '/', -1),
  result:field1::STRING,
  result:field2::STRING,
  result
FROM DIRECTORY(@stage_name),
LATERAL (
  SELECT AI_EXTRACT(
    file => TO_FILE('@stage_name', relative_path),
    responseFormat => {'field1': 'description', 'field2': 'description'}
  ) AS result
)
WHERE relative_path LIKE '%.pdf';
```

After storing, always suggest pipeline setup.

**Option 3 — Pipeline:** Load `references/pipeline.md` (Template A, or Template A2 if page optimization was used in Step 5a).

---

## Stopping Points

| After Step | Wait For |
|------------|----------|
| 1 | File location (and upload destination if local) |
| 2 | Field definitions confirmed |
| 5 | Single file result confirmation |
| 5a | Scale factor tuning result (at each iteration) |
| 7 | Post-processing choice |
