# SeaweedFS Image Upload Integration

## Overview

This document describes the SeaweedFS image upload integration for the DETAILED Excel extraction pipeline. Images extracted from Excel files are now uploaded to SeaweedFS/S3-compatible storage and referenced via URLs instead of storing large base64 data in Milvus.

## Architecture

### Data Flow

```
Excel File
    ↓
DETAILED Pipeline Extraction
    ↓
Images extracted as base64
    ↓
Upload to SeaweedFS (new step)
    ↓
Replace base64 with URLs
    ↓
Generate embeddings
    ↓
Store in Milvus (with URLs, not base64)
```

### Pipeline Steps

1. **Parse Excel (DETAILED)** - `parse_excel_detailed`
   - Extracts images from Excel
   - Stores base64 data in chunk metadata
   - Output: chunks with `image_base64` in `_legacy_metadata.full_metadata`

2. **Upload Images** - `upload_images_to_seaweedfs` (NEW)
   - Processes chunks with `image_base64`
   - Uploads to SeaweedFS via S3 API
   - Replaces `image_base64` with `image_url` and `image_s3_key`
   - Output: updated chunks without base64, with URLs

3. **Generate Embeddings** - `generate_embeddings_chunks`
   - Receives chunks with image URLs
   - Generates embeddings
   - Output: embedded chunks

4. **Store in Milvus** - `store_vectors_task`
   - Stores vectors with metadata
   - Metadata includes `image_url` and `image_s3_key`
   - Base64 data is NOT stored (already removed)

## Components

### 1. SeaweedFSService (`src/services/seaweedfs_service.py`)

Core service for uploading images to S3-compatible storage.

#### Methods

**`upload_image()`**
- Uploads a single image to SeaweedFS
- Parameters:
  - `image_base64`: Base64-encoded image data
  - `document_id`: Document UUID
  - `image_number`: Sequential image number
  - `sheet_name`: Sheet name where image was found
  - `mime_type`: Image MIME type (default: "image/png")
  - `bucket`: S3 bucket name (default: "textiq-images")
- Returns: Dict with `image_s3_key`, `image_url`, `success`

**`upload_images_batch()`**
- Uploads all images from a batch of chunks
- Parameters:
  - `chunks`: List of chunk dicts
  - `document_id`: Document UUID
  - `bucket`: S3 bucket name
- Returns: Updated chunks with URLs instead of base64
- Side effect: Removes `image_base64` from metadata after successful upload

**`delete_document_images()`**
- Deletes all images for a document
- Parameters:
  - `document_id`: Document UUID
  - `bucket`: S3 bucket name
- Returns: Dict with `deleted_count`, `success`

#### S3 Key Structure

Images are stored with hierarchical keys:
```
documents/{document_id}/images/{sheet_name}/image_{number}.{ext}
```

Example:
```
documents/550e8400-e29b-41d4-a716-446655440000/images/Sheet1/image_1.png
documents/550e8400-e29b-41d4-a716-446655440000/images/Data/image_2.jpg
```

### 2. Upload Task (`upload_images_to_seaweedfs`)

Airflow task that orchestrates the upload process.

**Location:** `airflow/dags/tasks/processing_tasks.py`

**Execution:**
- Runs in external Python environment (`/opt/airflow/.venv`)
- Timeout: 15 minutes
- Triggered after `parse_excel_detailed`

**Process:**
1. Loads chunks from pickle file
2. Counts images with `image_base64`
3. Uploads images using `SeaweedFSService`
4. Updates chunks with URLs
5. Saves updated chunks back to pickle
6. Returns metadata with upload stats

**Return Value:**
```python
{
    'uploaded_count': 5,        # Number successfully uploaded
    'total_images': 5,          # Total images found
    'success': True,            # Overall success flag
    'data_file': '/path/...',   # Updated pickle file
    'document_id': 'uuid',      # Pass through
    'filename': 'test.xlsx',    # Pass through
    'chunk_count': 20,          # Total chunks
}
```

### 3. DAG Integration

**File:** `airflow/dags/document_extraction_dag.py`

**Task Order for Excel DETAILED Pipeline:**
```python
parse_excel_detailed
    ↓
upload_images_to_seaweedfs  # NEW TASK
    ↓
generate_embeddings_chunks
    ↓
merge_pipeline_results
    ↓
store_vectors_task
```

## Configuration

### Environment Variables

Required environment variables (set in docker-compose or .env):

```bash
# SeaweedFS/S3 Configuration
AWS_ACCESS_KEY_ID=dummy
AWS_SECRET_ACCESS_KEY=dummy
AWS_ENDPOINT_URL=http://seaweedfs:8333

# Alternative using SEAWEEDFS_* vars
SEAWEEDFS_HOST=localhost
SEAWEEDFS_PORT=8333
SEAWEEDFS_ACCESS_KEY_ID=dummy
SEAWEEDFS_SECRET_ACCESS_KEY=dummy
```

### Bucket Setup

Default bucket: `textiq-images`

The bucket is automatically created on first upload if it doesn't exist.

### Image Overwrites

**Behavior when uploading the same file multiple times:**

Each time you process a document, images are stored at:
```
documents/{document_id}/images/{sheet_name}/image_{number}.{ext}
```

- **Different document_id** (normal case): New images created, no overwrites
- **Same document_id** (reprocessing): Images at same paths are **overwritten** by S3 `put_object`

This is the standard S3 behavior - uploading to an existing key replaces the object.

## Metadata Changes

### Before (with base64)

```json
{
  "_legacy_metadata": {
    "element_type": "Image",
    "full_metadata": {
      "image_base64": "iVBORw0KGgoAAAANSUhEU...",  // Large!
      "mime_type": "image/png",
      "image_number": 1,
      "position_key": "C5",
      "sheet_name": "Sheet1",
      "width": 800,
      "height": 600
    }
  }
}
```

### After (with URLs)

```json
{
  "_legacy_metadata": {
    "element_type": "Image",
    "full_metadata": {
      "image_url": "http://seaweedfs:8333/textiq-images/documents/.../image_1.png",
      "image_s3_key": "documents/550e8400-.../images/Sheet1/image_1.png",
      "mime_type": "image/png",
      "image_number": 1,
      "position_key": "C5",
      "sheet_name": "Sheet1",
      "width": 800,
      "height": 600
      // image_base64 removed!
    }
  }
}
```

### Milvus Metadata

The following fields are stored in Milvus (from `full_metadata`):

- `element_type`: "Image"
- `image_url`: Full URL to access image
- `image_s3_key`: S3 key for programmatic access
- `image_number`: Sequential number
- `position_key`: Excel cell reference
- `mime_type`: Image MIME type
- `width`, `height`: Image dimensions
- `cell_reference`, `from_cell`, `to_cell`: Position info

**NOT stored in Milvus:**
- `image_base64`: Filtered out (too large)
- `row_number`: Not applicable for images
- `has_symbols`: Not applicable for images

## Benefits

### 1. Reduced Milvus Storage

Base64-encoded images can be 10-100+ KB each. For documents with many images:
- **Before**: 100 images × 50 KB = 5 MB in Milvus per document
- **After**: 100 images × ~200 bytes = 20 KB in Milvus per document

**Result**: ~99% reduction in metadata size

### 2. Better Performance

- Faster Milvus queries (smaller metadata)
- Faster embedding generation (no large base64 in context)
- Efficient image retrieval via SeaweedFS

### 3. Scalability

- SeaweedFS designed for binary storage
- Milvus optimized for vectors and small metadata
- Clean separation of concerns

### 4. Image Management

- Easy to update/delete images independently
- Can implement image versioning
- Can serve images directly via HTTP

## Testing

### Unit Test

Run the SeaweedFS upload test:

```bash
cd /Users/nguyen/Work/textiq-doc-extraction
python scripts/test_seaweedfs_upload.py
```

Expected output:
```
======================================================================
SeaweedFS Image Upload Test
======================================================================

1. Creating test chunks with image data...
   ✓ Created 3 test chunks
   ✓ Found 2 images with base64 data

2. Test document ID: 550e8400-e29b-41d4-a716-446655440000

3. Initializing SeaweedFS service...
   ✓ SeaweedFS service initialized

4. Uploading images to SeaweedFS...
   ✓ Upload batch completed

5. Verifying upload results...

   Image chunk 2:
      - image_base64 removed: ✓ YES
      - image_s3_key added: ✓ YES
        → documents/550e8400-.../images/Sheet1/image_1.png
      - image_url added: ✓ YES
        → http://seaweedfs:8333/textiq-images/documents/.../image_1.png

   Image chunk 3:
      - image_base64 removed: ✓ YES
      - image_s3_key added: ✓ YES
        → documents/550e8400-.../images/Data/image_2.png
      - image_url added: ✓ YES
        → http://seaweedfs:8333/textiq-images/documents/.../image_2.png

======================================================================
Upload Summary: 2/2 images successfully uploaded
✓ All images uploaded successfully!
======================================================================
```

### Integration Test

Test the full pipeline with a real Excel file:

```bash
# Upload an Excel file with images
python scripts/upload_documents.py /path/to/excel_with_images.xlsx

# Check Airflow logs for upload task
# Check Milvus metadata for image_url fields
# Verify images are accessible via SeaweedFS URLs
```

## Troubleshooting

### Upload Fails

**Symptoms:**
- Task fails with S3 connection error
- Logs show "Failed to upload image"

**Solutions:**
1. Check SeaweedFS is running:
   ```bash
   docker ps | grep seaweedfs
   ```

2. Verify environment variables:
   ```bash
   docker exec airflow-worker env | grep AWS
   ```

3. Check network connectivity:
   ```bash
   docker exec airflow-worker curl http://seaweedfs:8333
   ```

### Images Not Found

**Symptoms:**
- `image_base64` still present after upload
- `image_url` field missing

**Solutions:**
1. Check upload task logs in Airflow
2. Verify task executed successfully
3. Check returned `uploaded_count` matches `total_images`

### Milvus Contains Base64

**Symptoms:**
- Milvus metadata still contains large `image_base64` field

**Solutions:**
1. Ensure upload task runs BEFORE embeddings task
2. Check DAG task dependencies
3. Verify VectorStore filtering logic (should exclude `image_base64`)

## Future Enhancements

### 1. Image Optimization

Before upload, optimize images:
- Resize large images
- Convert to WebP format
- Generate thumbnails

### 2. CDN Integration

Serve images via CDN:
- CloudFront distribution
- Edge caching
- Geographic distribution

### 3. Image Deduplication

Detect duplicate images:
- Hash-based deduplication
- Save storage space
- Share references

### 4. Image Versioning

Track image changes:
- Version IDs in S3 keys
- Maintain history
- Rollback capability

## Related Documentation

- [Airflow Migration Tech Spec](./feats/airflow-migration-tech-spec.md)
- [Deep Dive: Airflow Extraction v2 Integration](./deep-dive-airflow-extraction-v2-integration.md)
- [Excel Parser Service](../src/extraction_v2/excel_parser_service.py)
- [Vector Store Implementation](../src/knowledge/vector_store.py)