# Sprint Change Proposal: Large Table Detection in ParseDocumentOperator

**Date:** 2025-12-10
**Status:** Pending Approval
**Triggered By:** Story 1.6 Code Analysis
**Scope Classification:** Minor

---

## Section 1: Issue Summary

### Problem Statement

The Airflow `ParseDocumentOperator` for Excel documents does NOT implement the large table detection and routing logic that exists in `ExtractionPipelineV2.process_document_with_large_table_detection()` method in `src/extraction_v2/pipeline.py`.

### Context

During implementation of Story 1.6 (Main DAG Definition), analysis revealed that the Excel processing path in Airflow operators differs from the production pipeline:

- **Current Airflow Implementation**: Simple Docling → HTML → extract tables flow
- **Production Pipeline (`pipeline.py`)**: Border detection → size classification → routes large tables (>1000 rows) to chunked processing, normal tables to standard Docling

### Evidence

1. `airflow/plugins/operators/parse_document.py` (lines 101-123): Uses only `DocumentToHtmlConverter` and `HtmlTableExtractor`
2. `src/extraction_v2/pipeline.py` (lines 406-505): Implements `process_document_with_large_table_detection` with:
   - `BorderTableDetector` for table boundary detection
   - Size classification (>1000 rows = large)
   - `LargeTableExtractor` for chunked processing
   - Result merging for large + normal tables

### Impact if Not Addressed

- Large Excel files (>1000 rows) will **timeout or crash** in Airflow
- Inconsistent behavior between Airflow and Celery processing
- Production failures when migrating traffic to Airflow (Epic 2)

---

## Section 2: Impact Analysis

### Epic Impact

| Epic | Impact | Details |
|------|--------|---------|
| Epic 1: Airflow Infrastructure | **Affected** | Story 1.3 needs enhancement |
| Epic 2: Traffic Migration | No change | Will benefit from fix |
| Epic 3: Celery Deprecation | No change | Unaffected |

### Story Impact

| Story | Status | Required Change |
|-------|--------|-----------------|
| Story 1.3 (Core Operators) | Done | Add acceptance criteria for large table detection |
| Story 1.6 (Main DAG) | Done | No change needed (DAG structure unchanged) |

### Artifact Conflicts

| Artifact | Conflict | Resolution |
|----------|----------|------------|
| Tech Spec | Minor | Update source file inventory to show `LargeTableExtractor` integrated into `ParseDocumentOperator` |
| Architecture | None | Already describes large table handling |
| PRD | None | FR6-10 Table Extraction already covers this |
| UI/UX | None | Backend processing only |

### Technical Impact

- **Files Modified**: 2 operators + 2 documentation files
- **New Dependencies**: None (reuses existing `LargeTableExtractor`)
- **Testing**: Add unit tests for large table detection path
- **CI/CD**: No changes needed

---

## Section 3: Recommended Approach

### Selected Path: Direct Adjustment

Modify `ParseDocumentOperator` to implement large table detection matching the `pipeline.py` flow.

### Rationale

1. **Low Effort**: Reuses existing, tested `LargeTableExtractor` code
2. **Low Risk**: No architectural changes, just enhancing existing operator
3. **No Timeline Impact**: Fits within current sprint capacity (4-5 hours)
4. **Maintains Consistency**: Ensures Airflow behavior matches Celery behavior

### Alternatives Considered

| Option | Verdict | Reason |
|--------|---------|--------|
| Create separate LargeTableOperator | Rejected | Adds complexity, requires DAG restructuring |
| Defer to Epic 2 | Rejected | Would cause production failures during traffic migration |
| Accept limitation | Rejected | Large files are common in production workload |

### Effort Estimate

| Task | Estimate |
|------|----------|
| Modify ParseDocumentOperator | 3 hours |
| Update ChunkContentOperator | 1 hour |
| Add unit tests | 1 hour |
| Update documentation | 15 min |
| **Total** | **~5 hours** |

### Risk Assessment

| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| Large table detection differs from pipeline | Low | Medium | Copy exact logic from `pipeline.py` |
| Performance regression | Low | Low | Use same chunked processing as production |
| Test coverage gaps | Low | Medium | Add specific tests for large table path |

---

## Section 4: Detailed Change Proposals

### Change 1: Modify ParseDocumentOperator (HIGH)

**File:** `airflow/plugins/operators/parse_document.py`

**Section:** Excel processing (lines 100-123)

**Current Implementation:**
```python
if self.doc_type == 'excel':
    # Excel → HTML conversion
    self.log.info("Converting Excel document to HTML")
    html = converter.convert_to_html(local_path)

    self.log.info("Extracting tables from HTML")
    extractor = HtmlTableExtractor()
    valid_tables, raw_text_tables = extractor.extract_all_tables(html)

    return {
        'doc_type': 'excel',
        'html_content': html,
        'tables': [{'html': t.html, 'rows': t.rows} for t in valid_tables],
        'raw_text_tables': [{'rows': t.rows} for t in raw_text_tables],
        ...
    }
```

**Proposed Implementation:**
```python
if self.doc_type == 'excel':
    from src.extraction_v2.large_table_extractor import LargeTableExtractor

    # Step 1: Detect tables via border detection
    self.log.info("Detecting tables via border detection")
    large_extractor = LargeTableExtractor()
    detected_tables = large_extractor.detect_tables(local_path)

    large_tables = [t for t in detected_tables if t.get('is_large', False)]
    normal_tables = [t for t in detected_tables if not t.get('is_large', False)]

    self.log.info(f"Table classification: {len(large_tables)} large, {len(normal_tables)} normal")

    all_tables = []
    all_raw_text_tables = []
    large_table_records = []
    html = None

    # Step 2: Process large tables with chunked extraction
    if large_tables:
        self.log.info(f"Processing {len(large_tables)} large table(s) with chunked extraction")
        for table in large_tables:
            result = large_extractor.extract_table(local_path, table)
            if result.success:
                large_table_records.extend(result.records)
                self.log.info(f"Large table: {result.rows_processed} rows from {result.sheet_name}")

    # Step 3: Process normal tables with Docling
    if normal_tables or not detected_tables:
        self.log.info("Converting document to HTML with Docling")
        html = converter.convert_to_html(local_path)

        if html:
            self.log.info("Extracting tables from HTML")
            extractor = HtmlTableExtractor()
            valid_tables, raw_text_tables = extractor.extract_all_tables(html)
            all_tables = [{'html': t.html, 'rows': t.rows} for t in valid_tables]
            all_raw_text_tables = [{'rows': t.rows} for t in raw_text_tables]

    return {
        'doc_type': 'excel',
        'html_content': html,
        'markdown_content': None,
        'tables': all_tables,
        'raw_text_tables': all_raw_text_tables,
        'large_table_records': large_table_records,
        'local_path': local_path,
        'document_id': document_id,
    }
```

**Justification:** Matches `process_document_with_large_table_detection` flow in `pipeline.py:406-505`. Large tables (>1000 rows) get efficient chunked processing.

---

### Change 2: Update ChunkContentOperator (MEDIUM)

**File:** `airflow/plugins/operators/chunk_content.py`

**Section:** Table chunking strategy

**Current Implementation:**
```python
if self.chunking_strategy == 'table':
    parse_result = context['ti'].xcom_pull(task_ids=self.parse_task_id)
    # Process tables only...
```

**Proposed Addition:**
```python
if self.chunking_strategy == 'table':
    parse_result = context['ti'].xcom_pull(task_ids=self.parse_task_id)

    # Handle pre-processed large table records
    large_table_records = parse_result.get('large_table_records', [])
    if large_table_records:
        self.log.info(f"Including {len(large_table_records)} pre-processed large table records")
        all_records.extend(large_table_records)

    # Process normal tables with header detection (existing logic)
    for table_data in parse_result.get('tables', []):
        # ... existing logic unchanged
```

**Justification:** Large tables are already processed with LLM header detection in `LargeTableExtractor`. Pass them through without re-processing.

---

### Change 3: Update Story 1.3 Documentation (LOW)

**File:** `docs/feats/airflow-migration/sprint_artifacts/story-1-3-core-operators.md`

**Section:** Acceptance Criteria

**Add:**
```markdown
- [x] `ParseDocumentOperator` for Excel implements large table detection matching `pipeline.py` flow
```

**Justification:** Documents the requirement that was implicit in the tech spec.

---

### Change 4: Update Tech Spec (LOW)

**File:** `docs/feats/airflow-migration-tech-spec.md`

**Section:** Source File Inventory (line 106)

**Change:**
```markdown
# OLD:
| `large_table_extractor.py` | Chunked large table processing | ~600 | Custom LargeTableOperator |

# NEW:
| `large_table_extractor.py` | Chunked large table processing | ~600 | ParseDocumentOperator (Excel, integrated) |
```

**Justification:** Clarifies that large table extraction is integrated into ParseDocumentOperator.

---

## Section 5: Implementation Handoff

### Change Scope Classification: Minor

This change can be implemented directly by the development team without requiring backlog reorganization or PM/Architect involvement.

### Handoff Recipients

| Role | Responsibility |
|------|---------------|
| **Dev Team** | Implement code changes and tests |
| **QA** | Verify large table handling works correctly |

### Implementation Tasks

1. [ ] Modify `parse_document.py` to add large table detection
2. [ ] Modify `chunk_content.py` to handle large table records
3. [ ] Add unit tests for large table detection path
4. [ ] Update story-1-3 acceptance criteria
5. [ ] Update tech spec source file inventory
6. [ ] Run full test suite
7. [ ] Test with sample large Excel file (>1000 rows)

### Success Criteria

- [ ] ParseDocumentOperator detects and classifies tables by size
- [ ] Large tables (>1000 rows) processed with chunked extraction
- [ ] Normal tables processed with standard Docling flow
- [ ] Results merged correctly in ChunkContentOperator
- [ ] All existing tests pass
- [ ] New tests for large table path pass
- [ ] Manual test with large Excel file succeeds

### Definition of Done

- Code changes merged to `seaweedfs-setup` branch
- All tests passing (existing + new)
- Documentation updated
- Verified in Docker Airflow environment

---

## Approval

- [x] **User Approval**: Approved (2025-12-10)
- [x] **Implementation Started**: Started (2025-12-10)
- [x] **Implementation Complete**: Complete (2025-12-10)

---

*Generated by Correct Course Workflow*
*Date: 2025-12-10*