# textiq-doc-extraction - Technical Specification

**Author:** TextIQ
**Date:** 2025-12-03
**Project Level:** Quick-flow (standalone)
**Change Type:** Feature Enhancement
**Development Context:** Brownfield - Extraction Pipeline V2

---

## Context

### Available Documents

- **Workflow Status:** BMad Method track (docs/bmm-workflow-status.yaml)
- **Existing PRD:** docs/prd.md
- **Architecture:** docs/architecture.md
- **Epics:** docs/epics.md
- **Sprint Status:** docs/sprint-artifacts/sprint-status.yaml

### Project Stack

| Component | Version | Purpose |
|-----------|---------|---------|
| Python | 3.11+ | Runtime |
| FastAPI | 0.122 | Web framework |
| Docling | 2.63.0+ | Document → HTML conversion |
| SQLAlchemy | 2.0+ | ORM |
| PostgreSQL | - | Database (via asyncpg) |
| Celery | 5.3+ | Task queue |
| OpenAI | 1.0+ | LLM for header detection |
| pytest | 8.0+ | Testing |
| ruff/black | - | Linting/formatting |

### Existing Codebase Structure

**V2 Extraction Pipeline Components:**

```
src/extraction_v2/
├── pipeline.py              # ExtractionPipelineV2 orchestrator
├── document_converter.py    # Excel → HTML via Docling
├── html_table_extractor.py  # HTML → HtmlTable objects
├── llm_header_detector.py   # LLM-based header detection
├── record_builder.py        # Table + Headers → Records
├── large_table_extractor.py # Chunked extraction for large tables
└── semantic_chunker.py      # Word document chunking
```

**Current Flow:**
1. Excel → HTML (Docling)
2. HTML → Tables (HtmlTableExtractor)
3. **Filter:** Tables with < 2 columns are discarded
4. Tables → LLM Header Detection
5. Tables + Headers → Records (RecordBuilder)

**Problem:** Step 3 discards single-column "tables" which contain raw text content from the Excel file.

---

## The Change

### Problem Statement

Docling wraps ALL Excel content in HTML `<table>` tags, including raw text (notes, descriptions, paragraphs) that aren't actual tabular data. The current pipeline filters out single-column tables (`MIN_TABLE_COLS = 2`), causing this raw text content to be lost.

Users need access to ALL content from Excel files, not just structured table data.

### Proposed Solution

Modify the V2 extraction pipeline to:
1. **Detect** single-column tables as "raw text" content
2. **Skip LLM header detection** for raw text (no headers needed, saves API calls)
3. **Build records directly** from raw text content
4. **Store** in existing `ExtractedRecord` model with `row_number = -1`

### Scope

**In Scope:**
- Modify `HtmlTableExtractor` to return both valid tables AND raw text tables (separately classified)
- Modify `ExtractionPipelineV2.process_document()` to handle raw text tables differently
- Create raw text records without LLM header detection
- Store raw text in `ExtractedRecord` using existing schema
- Unit tests for new functionality

**Out of Scope:**
- New database model or `record_type` field (deferred)
- Changes to query/search APIs
- Changes to Word document processing
- UI changes

---

## Implementation Details

### Source Tree Changes

| File | Action | Changes |
|------|--------|---------|
| `src/extraction_v2/html_table_extractor.py` | MODIFY | Add `extract_all_tables()` method returning `(valid_tables, raw_text_tables)` tuple |
| `src/extraction_v2/pipeline.py` | MODIFY | Update `process_document()` to process raw text tables without LLM |
| `src/extraction_v2/record_builder.py` | MODIFY | Add `build_raw_text_records()` method |
| `tests/unit/extraction_v2/test_html_table_extractor.py` | CREATE | Tests for raw text extraction |
| `tests/unit/extraction_v2/test_raw_text_records.py` | CREATE | Tests for raw text record building |

### Technical Approach

**1. HtmlTableExtractor Changes:**

Add classification method that separates tables into two categories:

```python
def extract_all_tables(self, html: str) -> tuple[list[HtmlTable], list[HtmlTable]]:
    """
    Extract all tables, classified as valid data tables or raw text.

    Returns:
        Tuple of (valid_tables, raw_text_tables)
        - valid_tables: Tables with >= MIN_TABLE_COLS columns
        - raw_text_tables: Single-column tables (raw text content)
    """
```

**2. Pipeline Changes:**

In `process_document()`, after extracting tables:

```python
# Extract and classify tables
valid_tables, raw_text_tables = self.extractor.extract_all_tables(html)

# Process valid tables with LLM header detection (existing flow)
for table in valid_tables:
    headers = self.detector.detect_headers(table.html)
    records = self.builder.build_records(table, headers, document_id, sheet_name)
    all_records.extend(records)

# Process raw text tables WITHOUT LLM (new flow)
for table in raw_text_tables:
    records = self.builder.build_raw_text_records(table, document_id, sheet_name)
    all_records.extend(records)
```

**3. RecordBuilder Changes:**

Add method to build records from raw text without headers:

```python
def build_raw_text_records(
    self,
    table: HtmlTable,
    document_id: UUID,
    sheet_name: str | None = None
) -> list[dict[str, Any]]:
    """
    Build records from single-column raw text table.

    Concatenates all rows into single record with row_number=0.
    No LLM header detection needed.
    """
```

### Existing Patterns to Follow

From `src/extraction_v2/record_builder.py`:
- Use structlog for logging
- Return `list[dict[str, Any]]` format
- Include `_source` metadata with `document_id`, `sheet`, `row`, `table_id`
- Filter empty records with `_is_record_non_empty()`

From `src/extraction_v2/html_table_extractor.py`:
- Use BeautifulSoup with 'lxml' parser
- Log extraction statistics
- Return `HtmlTable` dataclass objects

### Integration Points

- **Input:** HTML string from `DocumentToHtmlConverter`
- **Output:** Records added to `all_records` list in pipeline
- **Storage:** Existing `ExtractedRecord` model via database service
- **No changes** to downstream consumers (query API, vector indexing)

---

## Development Context

### Relevant Existing Code

| Location | Purpose |
|----------|---------|
| `src/extraction_v2/html_table_extractor.py:108-130` | `_is_valid_table()` - current filtering logic |
| `src/extraction_v2/html_table_extractor.py:50-106` | `extract_tables()` - extraction loop |
| `src/extraction_v2/pipeline.py:146-199` | Table processing loop in `process_document()` |
| `src/extraction_v2/record_builder.py:26-104` | `build_records()` method |
| `src/db/models.py:99-153` | `ExtractedRecord` model |

### Dependencies

**Framework/Libraries:**
- BeautifulSoup4 (via lxml) - HTML parsing
- structlog 24.0+ - Logging
- UUID from stdlib - Document IDs

**Internal Modules:**
- `src.extraction_v2.html_table_extractor.HtmlTable`
- `src.extraction_v2.llm_header_detector.HeaderStructure`
- `src.extraction_v2.record_builder.RecordBuilder`

### Configuration Changes

None required.

### Existing Conventions (Brownfield)

**Code Style:**
- Line length: 100 chars (black/ruff)
- Type hints: Yes (mypy strict)
- Imports: isort ordering
- Quotes: Double quotes
- Docstrings: Google style

**Test Patterns:**
- Framework: pytest with pytest-asyncio
- Location: `tests/unit/extraction_v2/`
- Naming: `test_*.py` files, `test_*` functions
- Fixtures: Defined in `conftest.py`

### Test Framework & Standards

- **Framework:** pytest 8.0+
- **Async:** pytest-asyncio with `asyncio_mode = "auto"`
- **Coverage:** No explicit threshold, but test critical paths
- **Assertions:** Standard pytest `assert`

---

## Implementation Stack

| Layer | Technology |
|-------|------------|
| Runtime | Python 3.11+ |
| HTML Parsing | BeautifulSoup4 + lxml |
| Logging | structlog |
| Testing | pytest, pytest-asyncio |
| Linting | ruff, black, mypy |

---

## Technical Details

**Raw Text Detection Logic:**

A table is considered "raw text" if:
- `col_count < MIN_TABLE_COLS` (currently MIN_TABLE_COLS = 2)
- In practice: single-column tables

**Record Structure for Raw Text:**

```json
{
  "content": {
    "text": "First row text\nSecond row text\nThird row text"
  },
  "_source": {
    "document_id": "uuid-string",
    "sheet": "Sheet1",
    "row": -1,
    "table_id": 0
  }
}
```

**Decision:** Use concatenated approach with `"text"` key for cleaner semantic meaning. `row_number = -1` distinguishes raw text from table records.

**Edge Cases:**
- Empty single-column tables: Skip (no record)
- Multiple single-column tables in same sheet: Create separate records, different `table_id`
- Mixed content (some valid tables, some raw text): Process both types

---

## Development Setup

```bash
# Clone and setup (if not already)
cd /Users/nguyen/Work/textiq-doc-extraction

# Install dependencies
uv sync

# Run tests
uv run pytest tests/unit/extraction_v2/ -v

# Run specific test file
uv run pytest tests/unit/extraction_v2/test_html_table_extractor.py -v

# Lint
uv run ruff check src/extraction_v2/
uv run black --check src/extraction_v2/
```

---

## Implementation Guide

### Setup Steps

1. Create feature branch: `git checkout -b feat/extract-raw-text`
2. Verify dev environment: `uv run pytest tests/unit/extraction_v2/ -v`
3. Review existing code at locations listed above

### Implementation Steps

**Step 1: Modify HtmlTableExtractor**

1. Add `extract_all_tables()` method that returns `(valid_tables, raw_text_tables)` tuple
2. Refactor `extract_tables()` to use new method internally (maintain backward compatibility)
3. Add logging for raw text table detection

**Step 2: Add RecordBuilder.build_raw_text_records()**

1. Add new method to build records from single-column tables
2. Concatenate all row text into single `"text"` field
3. Set `row_number = -1` in `_source` metadata
4. Skip empty tables

**Step 3: Update Pipeline**

1. Replace `extract_tables()` call with `extract_all_tables()`
2. Add processing loop for raw text tables (no LLM)
3. Extend `all_records` with raw text records
4. Update logging to include raw text counts

**Step 4: Write Tests**

1. Test `extract_all_tables()` classification
2. Test `build_raw_text_records()` output format
3. Test pipeline integration with mixed content
4. Test edge cases (empty tables, multiple raw text blocks)

### Testing Strategy

**Unit Tests:**
- `test_extract_all_tables_separates_valid_and_raw_text()`
- `test_single_column_table_classified_as_raw_text()`
- `test_multi_column_table_classified_as_valid()`
- `test_build_raw_text_records_concatenates_rows()`
- `test_build_raw_text_records_skips_empty()`
- `test_build_raw_text_records_sets_row_minus_one()`

**Integration Tests:**
- `test_pipeline_processes_mixed_content()`
- `test_pipeline_handles_only_raw_text()`
- `test_pipeline_handles_only_tables()`

### Acceptance Criteria

1. ✅ Single-column tables are extracted as raw text records
2. ✅ Raw text records do NOT trigger LLM header detection (cost savings)
3. ✅ Raw text stored in `ExtractedRecord` with `row_number = -1`
4. ✅ Content field contains concatenated text with `"text"` key
5. ✅ Existing table extraction continues to work unchanged
6. ✅ All unit tests pass
7. ✅ No ruff/black/mypy errors

---

## Developer Resources

### File Paths Reference

- `/Users/nguyen/Work/textiq-doc-extraction/src/extraction_v2/html_table_extractor.py`
- `/Users/nguyen/Work/textiq-doc-extraction/src/extraction_v2/pipeline.py`
- `/Users/nguyen/Work/textiq-doc-extraction/src/extraction_v2/record_builder.py`
- `/Users/nguyen/Work/textiq-doc-extraction/src/db/models.py`
- `/Users/nguyen/Work/textiq-doc-extraction/tests/unit/extraction_v2/`

### Key Code Locations

| Component | Location |
|-----------|----------|
| `HtmlTableExtractor._is_valid_table()` | `src/extraction_v2/html_table_extractor.py:108` |
| `HtmlTableExtractor.extract_tables()` | `src/extraction_v2/html_table_extractor.py:50` |
| `ExtractionPipelineV2.process_document()` | `src/extraction_v2/pipeline.py:90` |
| `RecordBuilder.build_records()` | `src/extraction_v2/record_builder.py:26` |
| `ExtractedRecord` model | `src/db/models.py:99` |

### Testing Locations

- Unit tests: `tests/unit/extraction_v2/`
- Manual tests: `tests/manual/`
- Existing tests: `tests/extraction/test_pipeline.py`

### Documentation to Update

- None required for this change

---

## UX/UI Considerations

No UI/UX impact - backend/API/infrastructure change only.

---

## Testing Approach

**Test Framework:** pytest 8.0+ with pytest-asyncio

**Test Strategy:**
- Unit tests for each modified component
- Integration test for full pipeline with mixed content
- Use fixtures for HTML samples with single-column and multi-column tables

**Coverage:**
- All new methods must have tests
- Edge cases: empty tables, multiple raw text blocks, mixed content

---

## Deployment Strategy

### Deployment Steps

1. Merge PR to main branch
2. CI/CD pipeline runs tests automatically
3. Deploy to staging environment
4. Verify extraction works with test Excel files containing raw text
5. Deploy to production
6. Monitor logs for raw text extraction counts

### Rollback Plan

1. Revert commit if issues found
2. Previous behavior (filtering single-column tables) is restored automatically
3. No data migration needed - this is additive

### Monitoring

- Log raw text table counts: "Found X raw text tables"
- Monitor LLM API call reduction (raw text skips LLM)
- Check record counts include raw text content