# Sprint Change Proposal: Docling-Based Extraction V2

**Date:** 2025-11-28
**Project:** DSOL - Document Processing Service
**Proposed By:** TextIQ
**Change Scope:** Moderate (Epic 3 enhancement)

---

## 1. Issue Summary

### Problem Statement
Need to implement **extraction_v2** using Docling library for Excel-to-HTML conversion with LLM-based header detection.

### Context
During sprint planning/implementation phase, identified opportunity to enhance table extraction using:
- **Docling** for Excel → HTML conversion (better structure preservation)
- **LLM intelligence** for header detection from first 10 rows
- **Hierarchical header mapping** (format: "Header 1 -> Sub Header -> Sub Sub Header")
- **JSON output** with mapped header-value pairs

### Discovery Method
Proactive enhancement identified by development team to improve extraction accuracy and handle complex table layouts.

---

## 2. Impact Analysis

### Epic Impact

#### **Epic 3: Extraction Pipeline** - MAJOR ENHANCEMENT
**Affected Stories:**
- ✅ **Story 3.1** (Table Boundary Detection): Enhanced with Docling HTML extraction
- ✅ **Story 3.2** (Multi-Level Header Extraction): Enhanced with LLM-based detection
- ✅ **Story 3.4** (Record Builder): Modified to work with HTML-extracted tables
- ✅ **Story 3.4a** (Pipeline Orchestrator): Updated to support extraction_v2
- ➕ **NEW Story 3.4b**: Docling-Based Excel to HTML Converter

**Unchanged Stories:**
- ✅ Story 3.5, 3.6 (Symbol Detection/Resolution) - still applicable
- ✅ Story 3.8, 3.8a (Cross-Reference Detection) - still applicable
- ✅ Multi-pass architecture - preserved

#### **Epic 4: Knowledge Base** - NO IMPACT
- Same JSON structure output → no changes needed

#### **Epic 5: Q&A** - NO IMPACT
- Retrieval and Q&A work with same data structures

### Artifact Conflicts

#### **PRD Alignment: ✅ NO CONFLICTS**
- **FR6-10** (Table Extraction): Still fully satisfied
- **FR7** (Multi-level headers): Actually IMPROVED with LLM intelligence
- MVP goals: Enhanced, not changed

#### **Architecture Updates: 🔧 MODERATE CHANGES**

**Required Updates:**
1. **Technology Stack** (architecture.md lines 148-177):
   - Add Docling as primary extraction tool
   - Keep openpyxl as fallback/validation

2. **Dependencies** (architecture.md line 444):
   - Add: Docling, beautifulsoup4, lxml

3. **Module Structure** (architecture.md lines 60-78):
   - Add new directory: `src/extraction_v2/` (V2 pipeline - Docling-based)
   - Add: `src/extraction_v2/pipeline.py` - V2 orchestrator
   - Add: `src/extraction_v2/excel_to_html_converter.py`
   - Add: `src/extraction_v2/html_table_extractor.py`
   - Add: `src/extraction_v2/llm_header_detector.py`
   - Add: `src/extraction_v2/record_builder.py`
   - Keep: `src/extraction/` (V1 pipeline - openpyxl-based, fallback)

**No Architectural Pattern Changes:**
- ✅ Still Pipeline Architecture with API Gateway
- ✅ Multi-pass extraction preserved
- ✅ Same data flow: Upload → Extract → Index → Query

#### **UX/UI Specifications: N/A**
- Backend-only change, no UI impact

---

## 3. Recommended Approach

### **Selected Path: Option 1 - Direct Adjustment** ✅

**Strategy:** Add extraction_v2 as enhanced extraction method alongside existing approach

**Implementation Plan:**

1. **Add New Story 3.4b** - Docling Excel to HTML Converter
   - Implement Docling integration
   - HTML table extraction logic
   - LLM-based header detection

2. **Update Story 3.4a** - Pipeline Orchestrator
   - Add extraction_v2 pipeline branch
   - Implement fallback to original extraction if Docling fails
   - A/B testing capability

3. **Update Existing Stories** (3.1, 3.2, 3.4)
   - Modify acceptance criteria to include v2 approach
   - Update technical notes with new modules

4. **Update Architecture Document**
   - Add Docling to technology stack
   - Document new extraction modules
   - Update dependencies list

### Why This Approach?

✅ **Advantages:**
- Non-breaking change (original extraction stays as fallback)
- Enables A/B testing between extraction methods
- Enhances extraction accuracy without risk
- Can ship incrementally

✅ **Risk Assessment:** LOW
- Fallback mechanism ensures no regression
- Incremental rollout possible

✅ **Timeline Impact:** Minimal
- +1 new story (3.4b)
- ~3-5 days implementation
- No delays to overall MVP timeline

---

## 4. Detailed Change Proposals

### **CHANGE 1: Architecture Document**

**File:** `docs/architecture.md`

**Edit 1.1 - Technology Stack (line ~150)**

```markdown
OLD:
**openpyxl**
- Full .xlsx support, merged cells

NEW:
**Docling**
- Excel to HTML conversion with table preservation
- Superior handling of complex table layouts

**openpyxl** (fallback)
- Metadata extraction and validation
- Fallback for Docling conversion failures
```

**Edit 1.2 - Dependencies (after line 444)**

```markdown
ADD:
- Docling for HTML-based Excel extraction
- BeautifulSoup4 for HTML parsing
- lxml for fast HTML processing
```

**Rationale:** Documents new primary extraction approach while maintaining backward compatibility.

---

### **CHANGE 2: Epic 3 Stories**

**File:** `docs/epics.md`

**Edit 2.1 - Story 3.1 Technical Notes (line ~630)**

```markdown
OLD:
**Technical Notes:**
- Use heuristics: header row has text in most cells, data rows have consistent patterns
- Consider using LLM for ambiguous cases (send first 20 rows for classification)
- Module: `src/extraction/table_detector.py`

NEW:
**Technical Notes:**
- **Primary approach (v2):** Use Docling to convert Excel → HTML, extract <table> elements
- **Fallback approach (v1):** Use openpyxl with heuristic detection
- LLM analyzes first 10 rows to detect header boundaries
- Modules:
  - V2: `src/extraction/excel_to_html_converter.py`, `src/extraction/html_table_extractor.py`
  - V1: `src/extraction/table_detector.py`
```

**Edit 2.2 - Story 3.2 Technical Notes (line ~716)**

```markdown
OLD:
**Technical Notes:**
- Detect header depth by counting rows before data starts
- Use `>` as separator in flattened names
- Module: `src/extraction/header_extractor.py`

NEW:
**Technical Notes:**
- **V2 approach:** LLM analyzes first 10 rows of HTML table to detect multi-level headers
- Use `->` as separator in hierarchical names (format: "Header 1 -> Sub Header -> Sub Sub Header")
- Handles 1-N levels of nesting dynamically
- Modules:
  - V2: `src/extraction/llm_header_detector.py`
  - V1: `src/extraction/header_extractor.py` (fallback)
```

**Edit 2.3 - Story 3.4 Technical Notes (line ~808)**

```markdown
ADD:
**V2 Integration:**
- Input: HTML table structure from Docling conversion
- LLM-detected headers map to cell values
- Output: JSON with hierarchical header keys
```

**Edit 2.4 - Story 3.4a Pipeline Steps (line ~825)**

```markdown
OLD:
**PASS 2 - Table Extraction (existing 3.4a flow):**
4. Executes existing pipeline: TableDetector → HeaderExtractor → MergedCellResolver → RecordBuilder

NEW:
**PASS 2 - Table Extraction:**
4. **V2 Pipeline (primary):**
   - ExcelToHtmlConverter (Docling) → HtmlTableExtractor → LlmHeaderDetector → RecordBuilder
   - On failure: Fall back to V1 pipeline
5. **V1 Pipeline (fallback):**
   - TableDetector → HeaderExtractor → MergedCellResolver → RecordBuilder
```

---

### **CHANGE 3: Add New Story 3.4b**

**File:** `docs/epics.md`

**Location:** After Story 3.4a (line ~882)

```markdown
### Story 3.4b: Docling-Based Excel to HTML Converter

As a **system**,
I want **to convert Excel files to HTML using Docling**,
So that **I can leverage HTML table structure for more robust extraction**.

**Acceptance Criteria:**

**Given** an uploaded Excel document
**When** the extraction pipeline triggers conversion
**Then** the system:
1. Uses Docling library to convert .xlsx/.xls to HTML
2. Preserves table structures as <table> elements
3. Maintains cell relationships (merged cells, spans)
4. Outputs clean HTML suitable for parsing

**And** handles conversion failures gracefully:
- Corrupted Excel files → log error, return None, trigger V1 fallback
- Unsupported Excel features → log warning, continue with available content
- Empty workbooks → return empty HTML structure

**And** conversion performance targets:
- ≤10 seconds for typical Excel file (< 10MB)
- Streams large files to avoid memory issues
- Logs conversion time for monitoring

**And** HTML table extraction:
- Identifies all <table> elements in HTML
- Extracts table structure (rows, cells, spans)
- Sends first 10 rows to LLM for header detection
- Returns structured table data with detected headers

**And** LLM header detection:
- Analyzes first 10 rows to identify header rows
- Detects multi-level headers (parent-child relationships)
- Outputs hierarchical header format: "Header 1 -> Sub Header -> Sub Sub Header"
- Handles 1-N levels of nesting

**Prerequisites:** Story 2.2 (Single Document Upload)

**Technical Notes:**
- Install dependencies:
  - `docling>=1.0.0` (Excel to HTML conversion)
  - `beautifulsoup4>=4.12.0` (HTML parsing)
  - `lxml>=5.0.0` (fast HTML parser backend)
- Modules:
  - `src/extraction/excel_to_html_converter.py` - Docling wrapper
  - `src/extraction/html_table_extractor.py` - HTML table parsing
  - `src/extraction/llm_header_detector.py` - LLM-based header detection
- Integration point: `src/extraction/pipeline.py` (add v2 branch)
- Fallback: If Docling fails, fall back to openpyxl-based extraction (Story 3.1)
- LLM prompt template stored in `src/llm/prompts.py`

**FR Coverage:** FR6 (Table boundary detection), FR7 (Multi-level headers), FR1 (Excel ingestion)

**Estimated Effort:** 3-5 days (Medium complexity)
```

---

### **CHANGE 4: Update pyproject.toml**

**File:** `pyproject.toml` (to be created in Story 1.1)

**Section:** [tool.poetry.dependencies]

```toml
ADD:
docling = "^1.0.0"  # Excel to HTML conversion
beautifulsoup4 = "^4.12.0"  # HTML parsing
lxml = "^5.0.0"  # HTML parser backend
```

**Rationale:** Required dependencies for extraction_v2 implementation.

---

## 5. Implementation Handoff

### Change Scope Classification: **MODERATE**

**Reason:**
- Adds 1 new story to existing epic
- Modifies 4 existing stories (acceptance criteria updates)
- Updates 2 documentation files
- No breaking changes to architecture
- Enhances existing functionality

### Handoff Recipients

**Primary:** Development Team (direct implementation)

**Responsibilities:**
1. ✅ Implement Story 3.4b (Docling converter)
2. ✅ Update Stories 3.1, 3.2, 3.4, 3.4a (integration changes)
3. ✅ Update architecture.md with new technology stack
4. ✅ Update epics.md with story changes
5. ✅ Add dependencies to pyproject.toml
6. ✅ Test both V1 (fallback) and V2 (primary) extraction paths

**Secondary:** Product Owner (for awareness)

**Responsibilities:**
1. 📋 Review updated epic document
2. 📋 Approve moderate scope increase (+1 story)
3. 📋 Confirm MVP timeline still achievable

### Success Criteria

**Implementation Complete When:**
- ✅ Story 3.4b implemented and tested
- ✅ Extraction pipeline supports both V1 and V2
- ✅ Fallback mechanism works (V2 fails → V1 runs)
- ✅ Architecture and epic documents updated
- ✅ All acceptance criteria for modified stories still pass
- ✅ Integration tests pass for both extraction paths

**Quality Gates:**
- ✅ V2 extraction accuracy ≥ V1 accuracy (measured on test dataset)
- ✅ Conversion time ≤ 10 seconds per file
- ✅ Graceful fallback on Docling failures
- ✅ LLM header detection accuracy ≥ 85% on test cases

---

## 6. Risk Assessment & Mitigation

| Risk | Severity | Mitigation |
|------|----------|------------|
| Docling library instability | Medium | Fallback to V1 extraction ensures no regression |
| LLM header detection inaccuracy | Medium | Fine-tune prompts, validate on test dataset |
| Performance degradation | Low | Set timeout limits, optimize HTML parsing |
| Integration complexity | Low | Incremental rollout, thorough testing |

**Overall Risk Level:** LOW (due to fallback mechanism)

---

## 7. Timeline Impact

**Original Timeline:** Epic 3 = 11 stories
**Updated Timeline:** Epic 3 = 12 stories (+1)

**Estimated Additional Effort:**
- Story 3.4b: 3-5 days
- Documentation updates: 1 day
- Integration testing: 1-2 days

**Total Additional Time:** ~5-8 days

**Impact on MVP Delivery:** NONE (can be parallelized with other Epic 3 stories)

---

## 8. Open Questions

1. ✅ **Docling version:** Which version should we target? (Recommend: latest stable ≥ 1.0.0)
2. ✅ **LLM model:** Use same Azure OpenAI GPT-4o deployment for header detection? (Recommend: Yes, for consistency)
3. ✅ **Fallback trigger:** When should system fall back to V1? (Recommend: On Docling exception or timeout > 15s)
4. ⏳ **A/B testing:** Should we run both pipelines in parallel for comparison? (Recommend: Yes, for first 2 weeks)

---

## Approval & Next Steps

**Proposal Status:** ⏳ Awaiting User Approval

**Next Steps After Approval:**
1. ✅ Update `docs/architecture.md` (CHANGE 1)
2. ✅ Update `docs/epics.md` (CHANGE 2 + CHANGE 3)
3. ✅ Create Story 3.4b in sprint backlog
4. ✅ Assign Story 3.4b to development team
5. ✅ Begin implementation

**Estimated Start:** Immediately after approval
**Estimated Completion:** +5-8 days from start

---

**🚀 Generated with BMAD Correct Course Workflow**
**Workflow Mode:** YOLO (Rapid Analysis)
**Confidence Level:** HIGH (low-risk enhancement with fallback)

---

## Appendix: Technical Implementation Notes

### Module Structure (New Files)

```python
# src/extraction_v2/excel_to_html_converter.py
class ExcelToHtmlConverter:
    """Converts Excel files to HTML using Docling."""

    def convert_to_html(self, file_path: str) -> str | None:
        """Convert Excel to HTML, return None on failure."""
        pass

# src/extraction_v2/html_table_extractor.py
class HtmlTableExtractor:
    """Extracts tables from HTML structure."""

    def extract_tables(self, html: str) -> List[HtmlTable]:
        """Parse HTML and extract all <table> elements."""
        pass

# src/extraction_v2/llm_header_detector.py
class LlmHeaderDetector:
    """Uses LLM to detect headers from first 10 rows."""

    def detect_headers(self, first_10_rows: List[List]) -> HeaderStructure:
        """Analyze rows and return hierarchical header structure."""
        pass

# src/extraction_v2/pipeline.py
class ExtractionPipelineV2:
    """V2 pipeline orchestrator using Docling."""

    async def process_document(self, document_id: UUID) -> ExtractionResult:
        """Process document using V2 pipeline."""
        pass
```

### LLM Prompt Template (Draft)

```python
# src/llm/prompts.py

HEADER_DETECTION_PROMPT = """
Analyze the first 10 rows of this table and identify the header row(s).

Table data:
{table_rows}

Tasks:
1. Identify which row(s) contain headers
2. Detect if headers are multi-level (spanning multiple rows)
3. Return hierarchical header structure in format: "Parent -> Child -> Subchild"

Return JSON:
{
  "header_rows": [1, 2],  // Which rows are headers
  "headers": [
    {"column": 0, "hierarchy": ["Product"]},
    {"column": 1, "hierarchy": ["Sales", "Q1"]},
    {"column": 2, "hierarchy": ["Sales", "Q2"]}
  ]
}
"""
```

---

_End of Sprint Change Proposal_