---
name: interactive-clustering
description: "Choose optimal clustering keys for interactive tables using query analysis and cardinality evaluation. Triggers: clustering key selection, pick clustering columns, optimize clustering, choose cluster by."
parent_skill: snowflake-interactive
---

# Clustering Key Selection for Interactive Tables

Systematic workflow for choosing the optimal clustering key using query pattern analysis and cardinality evaluation.

## When to Load

Load this skill when:
- User needs help choosing clustering columns
- Converting existing tables and unsure about clustering
- Queries are slow and clustering needs optimization
- User asks "what should I cluster by?"

---

## Overview

Clustering keys determine how Snowflake organizes data in micro-partitions. The right clustering key enables efficient pruning, dramatically improving query performance.

**Key principles:**
1. Cluster on columns most used in WHERE clauses
2. Order: lowest cardinality to highest cardinality
3. For date/timestamp: always truncate to day
4. Maximum 3-4 columns

---

## Workflow

### Step 1: Analyze Query Patterns

**Goal:** Identify which columns are most frequently filtered

#### Option A: Use Query History (Recommended)

If the table has existing query history, analyze pruning patterns:

```sql
-- Set time window for analysis
SET interval_start = DATEADD(day, -7, CURRENT_TIMESTAMP());
SET interval_end = CURRENT_TIMESTAMP();
SET target_table = 'MY_TABLE';
SET target_schema = 'MY_SCHEMA';
SET target_database = 'MY_DATABASE';

-- Analyze column-level pruning to find best clustering candidates
SELECT 
  column_name,
  access_type,
  SUM(num_queries) AS query_count,
  TO_NUMBER(SUM(aggregate_query_execution_time)/SUM(num_queries), 38, 1) AS avg_query_exec_time_ms,
  DIV0(SUM(partitions_scanned), SUM(num_queries)) AS avg_partitions_scanned_per_query,
  DIV0(SUM(partitions_pruned), SUM(partitions_pruned + partitions_scanned)) AS pruning_ratio,
  1 - (DIV0(SUM(rows_matched), SUM(rows_scanned + rows_pruned))) AS rows_unused_ratio,
  SUM(rows_scanned) - SUM(rows_matched) AS unused_read_rows
FROM SNOWFLAKE.ACCOUNT_USAGE.COLUMN_QUERY_PRUNING_HISTORY
WHERE interval_start_time BETWEEN $interval_start AND $interval_end
  AND database_name = $target_database
  AND schema_name = $target_schema
  AND table_name = $target_table
GROUP BY column_name, access_type
ORDER BY unused_read_rows DESC
LIMIT 10;
```

**What to look for:**
- High `query_count`: Column is frequently used
- High `rows_unused_ratio`: Column has poor pruning (needs better clustering)
- High `unused_read_rows`: Biggest opportunity for improvement

**Top candidates** are columns with:
- High query count (frequently filtered)
- High unused read rows (poor current pruning)
- Used in WHERE clauses (access_type = 'WHERE')

#### Option B: Analyze Query Text (Alternative)

If pruning history is not available or table is new:

```sql
-- Find columns most used in WHERE clauses from query history
SELECT 
  query_text,
  COUNT(*) AS query_count
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE query_text ILIKE '%FROM%' || $target_table || '%WHERE%'
  AND start_time >= DATEADD(day, -7, CURRENT_TIMESTAMP())
GROUP BY query_text
ORDER BY query_count DESC
LIMIT 20;
```

**Manually identify** which columns appear most frequently in WHERE clauses.

#### Option C: Ask User (Fallback)

If no query history exists:

**Ask** user:
```
What columns do you most frequently filter on in WHERE clauses?

Examples:
- customer_id
- order_date  
- event_timestamp
- region
- user_id

List 1-3 columns you use most often in WHERE conditions.
```

---

### Step 2: Evaluate Cardinality

For each candidate column, check cardinality (number of distinct values):

```sql
-- Check cardinality for candidate columns
SELECT 
  'column1' AS column_name,
  APPROX_COUNT_DISTINCT(column1) AS cardinality
FROM {{database}}.{{schema}}.{{table_name}}

UNION ALL

SELECT 
  'column2' AS column_name,
  APPROX_COUNT_DISTINCT(column2) AS cardinality
FROM {{database}}.{{schema}}.{{table_name}}

UNION ALL

SELECT 
  'column3' AS column_name,
  APPROX_COUNT_DISTINCT(column3) AS cardinality
FROM {{database}}.{{schema}}.{{table_name}}

ORDER BY cardinality;
```

**Cardinality guidelines:**

| Cardinality Range | Clustering Suitability | Notes |
|-------------------|------------------------|-------|
| < 10 | ⚠️ Very low | Minimal pruning benefit (e.g., true/false column) |
| 10 - 1,000 | ✅ Good | Ideal for first clustering column |
| 1,000 - 100,000 | ✅ Good | Suitable for multi-column clustering |
| 100,000 - 1M | ⚠️ Moderate | Use expressions to reduce (e.g., TRUNC for timestamps) |
| > 1M | ❌ Too high | Use expressions to reduce cardinality |

---

### Step 3: Apply Clustering Rules

#### Rule 1: Order by Cardinality (Low to High)

**Always order clustering columns from lowest to highest cardinality:**

```sql
-- ✅ CORRECT: region (low) before customer_id (high)
CLUSTER BY (region, customer_id)

-- ❌ INCORRECT: customer_id (high) before region (low)  
CLUSTER BY (customer_id, region)
```

**Why?** Snowflake clusters hierarchically. Lower cardinality first creates larger, more efficient groups.

#### Rule 2: Truncate Date/Timestamp Columns

**Always truncate timestamp columns to day-level precision:**

```sql
-- ✅ CORRECT: Truncate to day
CLUSTER BY (TRUNC(event_timestamp, 'day'))
CLUSTER BY (TO_DATE(order_date))

-- ❌ INCORRECT: Full timestamp (too high cardinality)
CLUSTER BY (event_timestamp)
```

**Why?** Queries typically filter by date ranges (last 7 days, last month), not specific timestamps. Day-level precision provides optimal pruning for date-range queries.

#### Rule 3: Reduce High-Cardinality Columns

For high-cardinality columns (> 100K distinct values), use expressions to reduce:

```sql
-- High-cardinality numeric ID: truncate to reduce precision
CLUSTER BY (TRUNC(order_id, -5))  -- Groups by 100K blocks

-- High-cardinality string: use substring if prefix varies
CLUSTER BY (SUBSTRING(transaction_id, 1, 10))

-- Geographic coordinates: round to fewer decimal places
CLUSTER BY (ROUND(latitude, 2), ROUND(longitude, 2))
```

#### Rule 4: VARCHAR First 5-6 Bytes Only

**Important:** Snowflake clustering on VARCHAR uses only the first 5-6 bytes.

```sql
-- If common prefix (e.g., "CUST_"), consider substring after prefix
CLUSTER BY (SUBSTRING(customer_id, 6, 10))  -- Skip "CUST_" prefix

-- Or if IDs are numeric strings, convert to number
CLUSTER BY (TRY_TO_NUMBER(customer_id))
```

#### Rule 5: Maximum 3-4 Columns

**Limit clustering key to 3-4 columns max:**

```sql
-- ✅ GOOD: 3 columns, well-ordered
CLUSTER BY (region, TO_DATE(order_date), product_category)

-- ⚠️ TOO MANY: 5+ columns increases cost without proportional benefit
CLUSTER BY (region, country, state, city, zip_code, order_date)
```

---

### Step 4: Default Heuristic (When Unsure)

**When query history is unavailable or ambiguous, use this heuristic:**

1. **Identify most-filtered columns** from typical queries
2. **For date/timestamp columns**: Always use `TO_DATE()` or `TRUNC(col, 'day')`
3. **Order by cardinality**: Low to high
4. **Start with 1-2 columns**, expand only if needed

**Common patterns:**

```sql
-- Time-series events
CLUSTER BY (TRUNC(event_timestamp, 'day'))

-- Customer orders
CLUSTER BY (customer_id, TO_DATE(order_date))

-- Multi-dimensional analytics  
CLUSTER BY (region, TO_DATE(sale_date))

-- IoT/sensor data
CLUSTER BY (TRUNC(timestamp, 'day'), sensor_id)
```

---

### Step 5: Generate Clustering Key Recommendation

**Present recommendation** with explanation:

```
Based on analysis of {{table_name}}:

Top filtered columns:
1. {{column1}} - {{query_count1}} queries, {{cardinality1}} distinct values
2. {{column2}} - {{query_count2}} queries, {{cardinality2}} distinct values  
3. {{column3}} - {{query_count3}} queries, {{cardinality3}} distinct values

Recommended clustering key:
CLUSTER BY ({{recommended_clustering_key}})

Reasoning:
- {{column1}}: Most frequently filtered, low cardinality ({{cardinality1}})
- {{column2}}: Second most filtered, {{reasoning2}}
- Ordering: Low to high cardinality for optimal pruning
```

**⚠️ MANDATORY STOPPING POINT**: Get user approval before using this clustering key.

---

## Complete Examples

### Example 1: Order Table

**Scenario:** Orders table with customer_id, order_date, region

```sql
-- Check cardinality
SELECT 
  'customer_id' AS col, APPROX_COUNT_DISTINCT(customer_id) AS card FROM orders
UNION ALL
SELECT 'region', APPROX_COUNT_DISTINCT(region) FROM orders
UNION ALL  
SELECT 'order_date', APPROX_COUNT_DISTINCT(order_date) FROM orders;

-- Results: region = 5, order_date = 365, customer_id = 50000

-- Recommended clustering (low to high cardinality)
CLUSTER BY (region, TO_DATE(order_date))
```

### Example 2: Event Stream

**Scenario:** High-volume event stream with timestamps and event types

```sql
-- Check cardinality
SELECT 
  'event_type' AS col, APPROX_COUNT_DISTINCT(event_type) AS card FROM events
UNION ALL
SELECT 'user_id', APPROX_COUNT_DISTINCT(user_id) FROM events
UNION ALL
SELECT 'event_timestamp', APPROX_COUNT_DISTINCT(event_timestamp) FROM events;

-- Results: event_type = 25, user_id = 2M, timestamp = 50M

-- Recommended clustering
-- 1. Truncate timestamp (too high cardinality)
-- 2. event_type is low cardinality
-- 3. Skip user_id (too high, not frequently filtered alone)
CLUSTER BY (event_type, TRUNC(event_timestamp, 'day'))
```

### Example 3: Sales Analytics

**Scenario:** Sales data with region, product_category, sale_date

```sql
-- All columns have good cardinality (10-1000 range)
-- Order by typical query pattern and cardinality

-- Queries typically filter: region first, then date range, then category
CLUSTER BY (region, TO_DATE(sale_date), product_category)
```

---

## Analyze Column Usage Together

Check if columns are used together or separately:

```sql
-- Analyze which columns are used together in queries
WITH column_agg AS (
  SELECT 
    query_hash,
    query_parameterized_hash,
    interval_start_time,
    ARRAY_AGG(column_name) WITHIN GROUP (ORDER BY column_name) AS column_list,
    ANY_VALUE(num_queries) AS num_queries,
    ANY_VALUE(partitions_scanned) AS partitions_scanned,
    ANY_VALUE(partitions_pruned) AS partitions_pruned
  FROM SNOWFLAKE.ACCOUNT_USAGE.COLUMN_QUERY_PRUNING_HISTORY
  WHERE interval_start_time BETWEEN $interval_start AND $interval_end
    AND database_name = $target_database
    AND schema_name = $target_schema
    AND table_name = $target_table
  GROUP BY query_hash, query_parameterized_hash, interval_start_time
)
SELECT 
  column_list,
  SUM(num_queries) AS query_count,
  TO_NUMBER(SUM(partitions_scanned) / SUM(num_queries), 38, 1) AS avg_partitions_scanned,
  DIV0(SUM(partitions_pruned), SUM(partitions_pruned + partitions_scanned)) AS pruning_ratio
FROM column_agg
GROUP BY column_list
ORDER BY query_count DESC
LIMIT 20;
```

**If columns are always used together**: Include both in clustering key  
**If columns are used separately**: Prioritize the most-used column

---

## Validation and Testing

### Before: Baseline Query Performance

Before changing clustering, establish baseline:

```sql
-- Turn off result cache for fair comparison
ALTER SESSION SET USE_CACHED_RESULT = FALSE;

-- Run test query 5 times, measure latency
SELECT 
  customer_id,
  COUNT(*) AS order_count,
  SUM(total_amount) AS total_sales
FROM {{table_name}}
WHERE order_date >= CURRENT_DATE() - 30
GROUP BY customer_id
LIMIT 100;
```

Record execution times: p50, p90, p99.

### After: Test New Clustering

```sql
-- Recreate table with new clustering
CREATE OR REPLACE INTERACTIVE TABLE {{table_name}}_new
CLUSTER BY ({{new_clustering_key}})
AS SELECT * FROM {{table_name}};

-- Test same query 5 times
-- Compare execution times
```

**If performance improves significantly (> 30%)**: Use new clustering  
**If performance similar or worse**: Revert to original clustering

---

## Common Issues

### "Clustering made queries slower"

**Cause:** Clustering key doesn't match query patterns

**Fix:**
- Verify clustering columns match WHERE clauses
- Check column order (low to high cardinality)
- Ensure date columns are truncated to day

### "Too many clustering columns"

**Symptom:** High clustering cost, minimal performance gain

**Fix:** Reduce to 2-3 most important columns

### "Queries on different columns equally common"

**Solution:** Choose clustering based on:
1. Query frequency (which runs more often?)
2. Query importance (which needs to be faster?)
3. Default: cluster on date column if time-series data

---

## Stopping Points

1. ✋ After query analysis (present findings)
2. ✋ After clustering recommendation (get approval)
3. ✋ After testing (verify performance improvement)

---

## Output

- Analyzed query patterns with pruning statistics
- Evaluated column cardinality
- Applied clustering rules (order, truncation, limits)
- Generated clustering key recommendation with reasoning
- User-approved clustering key ready for CREATE TABLE
