# Extraction Pipeline Review

**Date:** 2025-11-27
**Reviewer:** Amp (AI Assistant)
**Scope:** Review of `src/extraction` code and `docs/specs/extraction-pipeline-tech-spec.md` against Requirements in `docs/prd.md`.

## Executive Summary

The current extraction pipeline implementation (`src/extraction/`) and technical specification align well with the "Table-Aware Extraction" requirements (FR6-FR10) of the PRD. The system correctly handles table boundary detection, multi-level headers, and merged cell resolution using a robust single-pass architecture.

**CRITICAL FINDING:** The current architecture **does not support the "Symbol Resolution" requirements (FR11-FR14)** which are listed as Core MVP Features in the PRD. The PRD explicitly mandates a **multi-pass** extraction approach to handle symbol dictionaries, whereas the current implementation is **single-pass**.

## Critical Issues

### 1. Architecture Mismatch: Single-Pass vs. Multi-Pass
- **Requirement (PRD):** The PRD explicitly states: *"Why multi-pass matters: Symbol dictionaries must be found and loaded BEFORE processing content sheets. Single-pass extraction misses this dependency."* (Section 3, Symbol Resolution).
- **Implementation (`src/extraction/pipeline.py`):** The `process_document` method iterates through sheets linearly once (`for sheet_idx, sheet in enumerate(workbook.worksheets)`). There is no preliminary pass to identify or load symbol dictionaries.
- **Impact:** Any symbols (e.g., "○", "×") defined in separate dictionary sheets will NOT be resolved in the extracted records, failing FR11, FR12, and FR13.

### 2. Missing Symbol Resolution Components
- **Requirement (PRD):** FR11-FR14 require detecting dictionary sheets, building mappings, and resolving symbols in content.
- **Implementation:** The `src/extraction` module contains:
    - `TableDetector`
    - `HeaderExtractor`
    - `MergedCellResolver`
    - `RecordBuilder`
  It **lacks**:
    - `SymbolDictionaryDetector`
    - `SymbolResolver`
- **Status:** The Tech Spec notes "Stories 3.1-3.4 Complete". Symbol Resolution (Stories 3.5/3.6) is missing from the implemented pipeline.

### 3. Data Model Discrepancy
- **Tech Spec (`docs/specs/extraction-pipeline-tech-spec.md`):** Defines `ExtractedRecord` with a `resolved_content` field:
  ```python
  resolved_content: dict | None   # Optional: After symbol resolution (Epic 3, Story 3.6)
  ```
- **Code (`src/extraction/models.py`):** The `ExtractedRecord` dataclass **excludes** the `resolved_content` field entirely.
  ```python
  @dataclass
  class ExtractedRecord:
      content: dict
      headers: list[str]
      _source: dict
      _merge_metadata: Optional[dict] = None
      # resolved_content IS MISSING
  ```

## Observations & Recommendations

### 1. Refactor for Multi-Pass Processing
The `ExtractionPipeline.process_document` method needs to be refactored to implement the Multi-Pass strategy defined in the PRD:
- **Pass 1 (Symbol Discovery):** Iterate all sheets to identify "Symbol Dictionary" sheets (based on patterns described in PRD). Build a global symbol map for the document.
- **Pass 2 (Content Extraction):** Iterate content sheets to extract tables, then apply the symbol map to populate a `resolved_content` field.

### 2. Update Data Models
- Update `ExtractedRecord` in `src/extraction/models.py` to include `resolved_content` field as per the Tech Spec and PRD requirements.

### 3. Clarify Cross-Reference Handling
- **Requirement (PRD):** FR15/FR16 mentions detecting and resolving cross-references (e.g., "See Sheet X").
- **Current State:** No specific logic exists in the extraction pipeline to detect these patterns.
- **Recommendation:** Determine if "Cross-Reference Detection" should be a specific step in the `RecordBuilder` or if it is deferred entirely to the Indexing phase. If it's in Extraction, a new component or rule in `RecordBuilder` is needed.

## Conclusion

The extraction pipeline is solid foundationally but is **incomplete regarding MVP requirements**. It successfully implements the "physical" extraction of Excel data (tables, headers, merges) but fails to implement the "semantic" enrichment (symbols) required by the PRD. Immediate attention is needed to implement the multi-pass architecture before proceeding to Indexing/Q&A.