# Sprint Change Proposal: HTML Table Size Filtering

**Date:** 2025-11-28
**Project:** DSOL - Document Processing Service
**Proposed By:** TextIQ
**Change Scope:** Minor (Bug Fix / Enhancement)
**Related Story:** Story 3.4b (Docling-Based Excel to HTML Converter)

---

## 1. Issue Summary

### Problem Statement

The V2 extraction pipeline's HTML table extraction step is processing **non-table HTML elements** (raw text formatted as minimal HTML tables) as legitimate data tables, causing noise in extracted records.

### Context

**Discovery:** Identified during V2 pipeline testing/implementation
**When Discovered:** 2025-11-28, during extraction testing with customer documents
**Current Behavior:** `HtmlTableExtractor.extract_tables()` returns ALL `<table>` elements found in HTML, regardless of size
**Impact:** False positives - single-cell or minimal tables (1-2 rows/cols) are processed as data tables

### Evidence

**Example Non-Tables Being Processed:**
1. **Text blocks:** Single-cell tables containing paragraphs (1 row × 1 col)
2. **Formatting wrappers:** 2×1 tables used for layout (not data)
3. **Title sections:** Small tables with document metadata (2×2)

**Consequence:**
- Unnecessary LLM API calls for non-table elements (cost + latency)
- Noise in extracted records database
- Confusion in downstream processing (symbol detection, cross-references)

---

## 2. Impact Analysis

### Epic Impact

#### **Epic 3: Extraction Pipeline** - MINOR ENHANCEMENT
**Affected Story:**
- ✅ **Story 3.4b** (Docling-Based Excel to HTML Converter) - Add filtering logic

**Unchanged Stories:**
- All other Epic 3 stories remain unaffected
- No changes to V1 pipeline (openpyxl-based)

### Artifact Conflicts

#### **PRD Alignment: ✅ NO CONFLICTS**
- Still satisfies FR6 (Table boundary detection)
- Enhancement improves precision (fewer false positives)

#### **Architecture Updates: ✅ NO CHANGES NEEDED**
- Single module update (`src/extraction_v2/html_table_extractor.py`)
- No new dependencies
- No architectural pattern changes

#### **UX/UI Specifications: N/A**
- Backend-only change, no user-facing impact

### Technical Impact

**Module:** `src/extraction_v2/html_table_extractor.py`
**Method:** `HtmlTableExtractor.extract_tables(html: str) -> List[HtmlTable]`
**Change Type:** Add filtering logic before returning table list

---

## 3. Recommended Approach

### **Selected Path: Direct Adjustment** ✅

**Strategy:** Add size-based filtering to `extract_tables()` method to exclude non-tables

### Proposed Implementation

**Simplified Filtering:** Column-count only (minimum 2 columns)

**Filtering Criteria:**

```python
def is_valid_table(table: HtmlTable) -> bool:
    """
    Determine if HTML table is a legitimate data table.

    Filters out:
    - Single-column tables (text blocks, lists)

    Threshold:
    - Minimum columns: 2 (single-column is likely text, not tabular data)

    Rationale:
    - Many HTML conversions wrap plain text in single-column tables
    - Legitimate data tables have at least 2 columns (key-value pairs minimum)
    - Row count is not restricted (some valid tables may have few rows)
    """
    MIN_COLS = 2

    # Check column threshold
    if table.col_count < MIN_COLS:
        return False

    return True
```

**Integration Point:**

```python
class HtmlTableExtractor:
    def extract_tables(self, html: str) -> List[HtmlTable]:
        """Extract tables from HTML structure."""

        # ... existing extraction logic ...

        # NEW: Filter out non-tables
        valid_tables = [table for table in tables if self._is_valid_table(table)]

        logger.info(
            f"Filtered tables: {len(tables)} found, {len(valid_tables)} valid, "
            f"{len(tables) - len(valid_tables)} filtered out"
        )

        return valid_tables
```

### Why This Approach?

✅ **Advantages:**
- Simple, low-risk change (single method update)
- Immediate improvement in extraction quality
- Reduces unnecessary LLM API calls
- Configurable thresholds (can tune based on data)

✅ **Risk Assessment:** MINIMAL
- Pure filtering logic, no core extraction changes
- Can make thresholds configurable if needed
- Easy to test and validate

✅ **Timeline Impact:** NEGLIGIBLE
- 1-2 hours implementation
- 1 hour testing/validation
- Can be deployed immediately

---

## 4. Detailed Change Proposals

### **CHANGE 1: Update html_table_extractor.py**

**File:** `src/extraction_v2/html_table_extractor.py`

**Edit 1.1 - Add filtering method (after HtmlTable dataclass)**

```python
# Around line 30-40, after HtmlTable dataclass definition

@dataclass
class HtmlTable:
    """Extracted HTML table with metadata."""
    html: str
    rows: List[List[str]]
    sheet_name: Optional[str]
    table_index: int
    row_count: int
    col_count: int


# NEW METHOD - Add after HtmlTable dataclass
class HtmlTableExtractor:
    """Extracts tables from HTML structure with size filtering."""

    # Configuration constant
    MIN_TABLE_COLS = 2        # Minimum columns for valid table (filters single-column text)

    def _is_valid_table(self, table: HtmlTable) -> bool:
        """
        Determine if HTML table is a legitimate data table.

        Filters out:
        - Single-column tables (likely text blocks or lists, not data)

        Args:
            table: HtmlTable to validate

        Returns:
            True if table has at least 2 columns, False otherwise
        """
        # Check column threshold only
        if table.col_count < self.MIN_TABLE_COLS:
            logger.debug(
                f"Filtered table {table.table_index}: "
                f"single-column table ({table.col_count} < {self.MIN_TABLE_COLS}) "
                f"- likely text, not data"
            )
            return False

        return True
```

**Edit 1.2 - Update extract_tables method (around line 80-100)**

```python
OLD:
def extract_tables(self, html: str) -> List[HtmlTable]:
    """Extract tables from HTML structure."""

    # ... existing extraction logic ...

    logger.info(f"Found {len(tables)} table(s) in HTML")
    return tables


NEW:
def extract_tables(self, html: str) -> List[HtmlTable]:
    """Extract tables from HTML structure with size filtering."""

    # ... existing extraction logic ...

    # Filter out non-tables (text blocks, formatting tables)
    valid_tables = [table for table in tables if self._is_valid_table(table)]

    logger.info(
        f"Table extraction complete: {len(tables)} found, "
        f"{len(valid_tables)} valid, {len(tables) - len(valid_tables)} filtered out"
    )

    # Log details of filtered tables for debugging
    if len(tables) != len(valid_tables):
        filtered = [t for t in tables if not self._is_valid_table(t)]
        for table in filtered:
            logger.debug(
                f"Filtered table {table.table_index}: "
                f"{table.row_count}×{table.col_count} "
                f"(below minimum thresholds)"
            )

    return valid_tables
```

**Rationale:**
- Prevents non-table elements from consuming resources
- Improves extraction precision
- Maintains detailed logging for debugging/tuning

---

### **CHANGE 2: Update Story 3.4b Acceptance Criteria**

**File:** `docs/epics.md`

**Location:** Story 3.4b, Acceptance Criteria section (line ~260-280)

**Edit 2.1 - Add table filtering criteria**

```markdown
OLD:
**And** HTML table extraction:
- Identifies all <table> elements in HTML
- Extracts table structure (rows, cells, spans)
- Sends first 10 rows to LLM for header detection
- Returns structured table data with detected headers


NEW:
**And** HTML table extraction with filtering:
- Identifies all <table> elements in HTML
- **Filters out single-column tables** (text blocks, lists):
  - Minimum 2 columns (single-column is text, not tabular data)
- Extracts table structure (rows, cells, spans)
- Sends first 10 rows to LLM for header detection
- Returns structured table data with detected headers
- Logs filtered table count for monitoring
```

**Rationale:** Documents the filtering behavior in acceptance criteria for testing validation.

---

### **CHANGE 3: Add Configuration Constants**

**File:** `src/extraction_v2/html_table_extractor.py`

**Location:** Top of HtmlTableExtractor class (around line 25-30)

**Edit 3.1 - Add configurable thresholds**

```python
class HtmlTableExtractor:
    """
    Extracts tables from HTML structure.

    Table Filtering:
    - Excludes tables below minimum size thresholds
    - Prevents processing of text blocks formatted as tables
    - Configurable via class constants
    """

    # Table size thresholds (configurable)
    MIN_TABLE_ROWS = 3        # Minimum rows for valid table
    MIN_TABLE_COLS = 2        # Minimum columns for valid table
    MIN_TABLE_CELLS = 6       # Minimum total cells (rows × cols)

    def __init__(self, min_cols: int = None):
        """
        Initialize HTML table extractor.

        Args:
            min_cols: Override minimum column threshold (default: 2)
        """
        # Allow threshold override for testing/tuning
        if min_cols is not None:
            self.MIN_TABLE_COLS = min_cols

        logger.debug(
            f"HtmlTableExtractor initialized with column threshold: "
            f"cols≥{self.MIN_TABLE_COLS}"
        )
```

**Rationale:**
- Makes thresholds configurable for different document types
- Enables A/B testing of threshold values
- Supports future tuning based on production data

---

## 5. Implementation Handoff

### Change Scope Classification: **MINOR**

**Reason:**
- Single module update (1 file)
- Pure filtering logic (no core extraction changes)
- No new dependencies
- No architecture changes
- Low risk, high value

### Handoff Recipients

**Primary:** Development Team (immediate implementation)

**Responsibilities:**
1. ✅ Update `src/extraction_v2/html_table_extractor.py`:
   - Add `_is_valid_table()` method
   - Update `extract_tables()` to filter tables
   - Add configurable thresholds to `__init__()`
2. ✅ Update Story 3.4b acceptance criteria in `docs/epics.md`
3. ✅ Test filtering with sample documents
4. ✅ Verify LLM API call reduction
5. ✅ Monitor filtered table counts in production logs

**Testing Checklist:**
- [ ] Tables with ≥2 cols → Processed ✅
- [ ] Single-column tables (N×1) → Filtered out ❌
- [ ] Two-column tables (N×2) → Processed ✅
- [ ] Multi-column tables (N×3+) → Processed ✅
- [ ] Logging shows "X filtered out" message
- [ ] No regression in existing valid table extraction

### Success Criteria

**Implementation Complete When:**
- ✅ Filtering logic added and tested
- ✅ Configuration thresholds work correctly
- ✅ Logging shows filtered table counts
- ✅ Story 3.4b acceptance criteria updated
- ✅ No false negatives (valid tables filtered)
- ✅ Measurable reduction in LLM API calls

**Quality Gates:**
- ✅ Zero false negatives on test dataset (no valid tables filtered)
- ✅ >80% reduction in non-table processing
- ✅ LLM API call count reduced proportionally
- ✅ No increase in extraction errors

---

## 6. Risk Assessment & Mitigation

| Risk | Severity | Likelihood | Mitigation |
|------|----------|------------|------------|
| False negatives (valid tables filtered) | High | Very Low | Only filters single-column tables (very conservative) |
| Thresholds too aggressive | Low | Very Low | 2-column minimum is very conservative |
| Document-specific edge cases | Low | Low | Configurable threshold if needed |
| Regression in extraction | Low | Very Low | Pure filtering (no core logic changes) |

**Overall Risk Level:** MINIMAL

**Rollback Plan:** Remove filtering (revert to processing all tables) if false negatives detected

---

## 7. Timeline Impact

**Estimated Effort:**
- Implementation: 1-2 hours
- Testing: 1 hour
- Documentation: 30 minutes
- Code review: 30 minutes

**Total Time:** ~4 hours (0.5 day)

**Impact on MVP Delivery:** NONE (can be deployed immediately)

---

## 8. Metrics & Monitoring

**Before Implementation:**
- Track: Total tables extracted per document
- Track: LLM API calls per document
- Track: Extraction time per document

**After Implementation:**
- Track: Tables found vs. tables filtered
- Track: Reduction in LLM API calls (expected: 30-50%)
- Track: Extraction time improvement (expected: 10-20% faster)
- Monitor: False negative rate (target: 0%)

**Success Metrics:**
- ≥30% reduction in non-table processing
- 0% false negative rate (no valid tables filtered)
- ≥10% improvement in extraction performance

---

## 9. Alternative Approaches Considered

### Option 1: Heuristic-Based Filtering (Chosen ✅)
**Approach:** Simple row/column thresholds
**Pros:** Fast, simple, deterministic, configurable
**Cons:** May miss edge cases with unusual table sizes
**Decision:** Selected for simplicity and low risk

### Option 2: Content Analysis
**Approach:** Analyze cell content (text density, formatting)
**Pros:** More sophisticated detection
**Cons:** Higher complexity, slower, more edge cases
**Decision:** Rejected - overkill for current need

### Option 3: LLM-Based Classification
**Approach:** Ask LLM if HTML table is valid data table
**Pros:** Highest accuracy potential
**Cons:** Expensive (API cost), slow, defeats purpose of filtering
**Decision:** Rejected - contradicts goal of reducing LLM calls

---

## 10. Open Questions

1. ✅ **Threshold value:** Is 2 columns appropriate? (Recommend: Yes, very conservative - filters only single-column)
2. ⏳ **Configuration exposure:** Should threshold be in config file or env vars? (Recommend: Class constant for now, config later if needed)
3. ⏳ **Logging verbosity:** Log every filtered table or just summary? (Recommend: Summary at INFO, details at DEBUG)
4. ⏳ **Metrics dashboard:** Add filtered table count to monitoring? (Recommend: Yes, track reduction in LLM calls)

---

## Approval & Next Steps

**Proposal Status:** ⏳ Awaiting User Approval

**Next Steps After Approval:**
1. Update `src/extraction_v2/html_table_extractor.py` (CHANGE 1)
2. Update `docs/epics.md` Story 3.4b (CHANGE 2)
3. Test with sample documents
4. Commit changes to git
5. Deploy and monitor filtered table counts

**Estimated Start:** Immediately after approval
**Estimated Completion:** Same day (4 hours)

---

## Appendix A: Example Filtering Scenarios

### Scenario 1: Valid Data Table (Processed ✅)

```
Table: 5 rows × 4 cols
Row 0: ["Product", "Q1", "Q2", "Q3"]
Row 1: ["Widget A", "100", "120", "150"]
Row 2: ["Widget B", "80", "90", "95"]
Row 3: ["Widget C", "200", "210", "220"]
Row 4: ["Widget D", "50", "55", "60"]

Result: ✅ Passes threshold (4≥2 cols)
```

### Scenario 2: Single-Column Text Block (Filtered ❌)

```
Table: 1 row × 1 col
Row 0: ["This is a paragraph of text explaining something"]

Result: ❌ Fails column threshold (1<2) - single-column text
```

### Scenario 3: Small Two-Column Table (Processed ✅)

```
Table: 2 rows × 2 cols
Row 0: ["Label:", "Value"]
Row 1: ["Date:", "2025-11-28"]

Result: ✅ Passes threshold (2≥2 cols) - legitimate key-value table
```

### Scenario 4: Single-Column List (Filtered ❌)

```
Table: 10 rows × 1 col
Row 0: ["Item 1"]
Row 1: ["Item 2"]
...
Row 9: ["Item 10"]

Result: ❌ Fails column threshold (1<2) - likely a list, not tabular data
```

### Scenario 5: Edge Case - Very Small Valid Table (Processed ✅)

```
Table: 2 rows × 2 cols
Row 0: ["Name", "Score"]
Row 1: ["Alice", "95"]

Result: ✅ Passes threshold (2≥2 cols) - minimal but valid data table
```

---

## Appendix B: Configuration Options

### Option 1: Hardcoded Constants (Current Proposal)

```python
class HtmlTableExtractor:
    MIN_TABLE_COLS = 2  # Filter single-column tables only
```

**Pros:** Simple, fast, no config file needed
**Cons:** Requires code change to adjust

### Option 2: Environment Variables

```python
import os

class HtmlTableExtractor:
    MIN_TABLE_COLS = int(os.getenv("TABLE_MIN_COLS", 2))  # Default: 2 columns
```

**Pros:** Configurable without code change
**Cons:** Adds environment dependency

### Option 3: Config File (Future Enhancement)

```yaml
# config.yaml
extraction_v2:
  table_filtering:
    min_cols: 2  # Minimum columns (filter single-column tables)
```

**Pros:** Centralized configuration
**Cons:** More complex, overkill for now

**Recommendation:** Start with Option 1 (hardcoded), migrate to Option 2 if tuning needed in production

---

**🚀 Generated with BMAD Correct Course Workflow**
**Workflow Mode:** Incremental (Collaborative)
**Confidence Level:** HIGH (low-risk, high-value enhancement)

---

_End of Sprint Change Proposal_