# Story 3.6: Symbol Resolution in Content

Status: done

## Story

As a **system**,
I want **to resolve symbols in extracted content using detected dictionaries**,
So that **records have human-readable meanings alongside symbols**.

## Acceptance Criteria

1. **AC3.6.1:** Build in-memory symbol lookup from database
   - Load all symbols for a document from `symbol_dictionaries` table
   - Build dictionary mapping: `{symbol: meaning}`
   - Support document-scoped lookups (symbols specific to document)
   - Handle case-sensitive matching (preserve exact symbol strings)

2. **AC3.6.2:** Resolve symbols in extracted records
   - For each record in `extracted_records`:
     - Scan all field values for known symbols
     - Create `{field}_resolved` key with resolved meaning
     - Preserve original symbol value in original field
   - Example transformation:
     ```json
     BEFORE:
     {"content": {"Item Code": "2024-4_0019", "Screen": "○"}}

     AFTER:
     {
       "content": {"Item Code": "2024-4_0019", "Screen": "○"},
       "resolved_content": {
         "Item Code": "2024-4_0019",
         "Screen": "○",
         "Screen_resolved": "Applicable/Yes"
       }
     }
     ```

3. **AC3.6.3:** Handle multiple symbols per record
   - Resolve symbols appearing in any column value
   - Create `{field}_resolved` for each resolved symbol
   - Support multiple resolved fields per record

4. **AC3.6.4:** Handle unknown symbols gracefully
   - Log warning for symbols not in dictionary (structlog with context)
   - Leave unresolved symbols as-is in original content
   - Do NOT create `{field}_resolved` for unknown symbols
   - Track count of unresolved symbols in resolution summary

5. **AC3.6.5:** Update database with resolved content
   - Update `extracted_records.resolved_content` JSONB field
   - Use batch updates for performance (500 records/batch)
   - Preserve original `content` field unchanged

6. **AC3.6.6:** Return resolution summary
   - Total records processed
   - Total symbols resolved
   - Count of unresolved symbols (with list of unique unresolved symbols)
   - Any warnings or errors

## Tasks / Subtasks

- [x] **Task 1: Create SymbolResolver class** (AC: 3.6.1, 3.6.2)
  - [x] Create `src/extraction/symbol_resolver.py`
  - [x] Implement `SymbolResolver` class
  - [x] Implement `resolve_document(document_id: UUID, session: Session) -> SymbolResolutionSummary` main method
  - [x] Implement `_build_symbol_lookup(document_id: UUID, session: Session) -> dict[str, str]` helper

- [x] **Task 2: Implement symbol lookup builder** (AC: 3.6.1)
  - [x] Query `symbol_dictionaries` table for document_id
  - [x] Build `{symbol: meaning}` dictionary
  - [x] Handle empty dictionary (log info, return empty dict)
  - [x] Handle duplicate symbols (first wins, log warning)
  - [x] Return populated lookup dictionary

- [x] **Task 3: Implement record resolution** (AC: 3.6.2, 3.6.3)
  - [x] Implement `_resolve_record(record: ExtractedRecord, symbol_lookup: dict) -> dict | None` method
  - [x] Iterate all fields in `content` dictionary
  - [x] Check each field value against symbol_lookup
  - [x] Create `{field}_resolved` keys for matches
  - [x] Build `resolved_content` dictionary with all original fields + resolved fields
  - [x] Return resolved_content or None if no symbols resolved

- [x] **Task 4: Handle unknown symbols** (AC: 3.6.4)
  - [x] Track unknown symbols encountered during resolution
  - [x] Log warning per unknown symbol (first occurrence only to avoid spam)
  - [x] Include unknown symbols in resolution summary
  - [x] Do not create resolved fields for unknown symbols

- [x] **Task 5: Batch database updates** (AC: 3.6.5)
  - [x] Implement `_batch_update_resolved_content(updates: list[tuple[UUID, dict]], session: Session)` method
  - [x] Query records from `extracted_records` for document_id
  - [x] Build list of (record_id, resolved_content) tuples
  - [x] Update in batches of 500 using SQLAlchemy bulk_update_mappings
  - [x] Commit after each batch

- [x] **Task 6: Create data models** (AC: 3.6.6)
  - [x] Add `SymbolResolutionSummary` dataclass to models.py
    - Fields: document_id, records_processed, symbols_resolved, unresolved_count, unresolved_symbols, warnings
  - [x] Implement `to_dict()` method
  - [x] Add structured logging for summary

- [x] **Task 7: Unit tests** (AC: All)
  - [x] Create `tests/extraction/test_symbol_resolver.py`
  - [x] Test symbol lookup building from database
  - [x] Test single record resolution (symbol found)
  - [x] Test single record resolution (symbol not found)
  - [x] Test multiple symbols in one record
  - [x] Test batch update functionality
  - [x] Test empty symbol dictionary (no resolution performed)
  - [x] Test Unicode symbol handling (○, ×, △)

- [x] **Task 8: Integration tests** (AC: All)
  - [x] Test full flow: detect symbols → extract records → resolve symbols
  - [x] Test with real Excel file containing symbols
  - [x] Verify resolved_content stored correctly in database
  - [x] Test resolution summary accuracy

## Dev Notes

### Architecture Patterns and Constraints

From architecture.md:
- **Module location:** `src/extraction/symbol_resolver.py` (architecture.md:68)
- **Naming conventions:**
  - Module: `symbol_resolver.py` (snake_case)
  - Class: `SymbolResolver` (PascalCase)
  - Functions: `resolve_document()`, `_build_symbol_lookup()` (snake_case)
- **Error handling:** Use structured logging for warnings, not exceptions for non-critical issues
- **Logging:** structlog with context (document_id, symbols_resolved, unresolved_count)
- **Type hints:** Full type annotations required
- **Database:** Update `extracted_records.resolved_content` JSONB field

From epics.md Story 3.6:
- **Prerequisites:** Story 3.4 (Record Builder), Story 3.5 (Symbol Dictionary Detection)
- **Run as PASS 3:** Symbol resolution runs after records are extracted
- **Module:** `src/extraction/symbol_resolver.py`

### Learnings from Previous Story

**From Story 3.5: Symbol Dictionary Detection (Status: done)**

- **SymbolDetector available at:** `src/extraction/symbol_detector.py`
  - Use `SymbolDetector.detect_dictionaries(workbook)` to find dictionary sheets
  - Returns `list[DetectedDictionary]` with symbols extracted
  - Symbols stored in `symbol_dictionaries` table via `store_symbols()` method

- **Data Models Created in 3.5:**
  - `DetectedDictionary` dataclass in `src/extraction/models.py`
  - `SymbolEntry` dataclass with fields: symbol, meaning, context
  - `SymbolDetectionSummary` dataclass
  - All have `to_dict()` methods for serialization

- **Database Pattern Established:**
  - ORM model `SymbolDictionary` in `src/db/models.py`
  - Batch insert using `session.bulk_insert_mappings()`
  - Symbol lookup: `session.query(SymbolDictionary).filter_by(document_id=doc_id)`

- **Component Pattern:**
  - Class with main public method: `detect_dictionaries()`, `resolve_document()`
  - Use structlog with context parameters
  - Return dataclass results with `to_dict()` method
  - Handle errors gracefully, log warnings for non-critical issues

- **Testing Pattern:**
  - Unit tests in `tests/extraction/test_symbol_resolver.py`
  - Sample Excel files in `tests/fixtures/sample_excel/`
  - Test both positive and negative cases
  - Test edge cases (empty, unicode, duplicates)

[Source: docs/sprint-artifacts/3-5-symbol-dictionary-detection.md#Dev-Notes]

### Project Structure Notes

Expected module structure after Story 3.6:
```
src/extraction/
├── __init__.py
├── models.py              # ADD: SymbolResolutionSummary
├── table_detector.py      # Story 3.1
├── header_extractor.py    # Story 3.2
├── merged_cell_resolver.py # Story 3.3
├── record_builder.py      # Story 3.4
├── pipeline.py            # Story 3.4a
├── symbol_detector.py     # Story 3.5
└── symbol_resolver.py     # NEW: Story 3.6 - SymbolResolver class

tests/extraction/
├── test_symbol_detector.py    # Story 3.5
├── test_symbol_resolver.py    # NEW: Story 3.6 unit tests
└── test_integration.py        # MODIFY: Add resolution tests
```

### Database Schema Reference

From Story 1.2 (symbol_dictionaries - for reading):
```sql
CREATE TABLE symbol_dictionaries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    sheet_name VARCHAR(255),
    symbol VARCHAR(50) NOT NULL,
    meaning TEXT NOT NULL,
    context VARCHAR(255),
    created_at TIMESTAMP DEFAULT NOW()
);
```

From Story 1.2 (extracted_records - for updating):
```sql
CREATE TABLE extracted_records (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    sheet_name VARCHAR(255) NOT NULL,
    row_number INTEGER NOT NULL,
    col_range VARCHAR(50),
    content JSONB NOT NULL,           -- Original content (read)
    resolved_content JSONB,           -- After symbol resolution (UPDATE THIS)
    headers JSONB,
    created_at TIMESTAMP DEFAULT NOW()
);
```

### Resolution Algorithm

```python
def resolve_document(document_id: UUID, session: Session) -> SymbolResolutionSummary:
    # Step 1: Build symbol lookup
    symbol_lookup = _build_symbol_lookup(document_id, session)
    if not symbol_lookup:
        logger.info("no_symbols_to_resolve", document_id=str(document_id))
        return SymbolResolutionSummary(
            document_id=document_id,
            records_processed=0,
            symbols_resolved=0,
            unresolved_count=0,
            unresolved_symbols=[],
            warnings=[]
        )

    # Step 2: Load extracted records
    records = session.query(ExtractedRecord).filter_by(document_id=document_id).all()

    # Step 3: Resolve each record
    updates = []
    symbols_resolved = 0
    unresolved_symbols = set()

    for record in records:
        resolved_content, resolved_count, unresolved = _resolve_record(
            record.content, symbol_lookup
        )
        if resolved_content:
            updates.append((record.id, resolved_content))
            symbols_resolved += resolved_count
        unresolved_symbols.update(unresolved)

    # Step 4: Batch update database
    _batch_update_resolved_content(updates, session)

    # Step 5: Return summary
    return SymbolResolutionSummary(
        document_id=document_id,
        records_processed=len(records),
        symbols_resolved=symbols_resolved,
        unresolved_count=len(unresolved_symbols),
        unresolved_symbols=list(unresolved_symbols),
        warnings=[]
    )
```

### Record Resolution Logic

```python
def _resolve_record(
    content: dict,
    symbol_lookup: dict[str, str]
) -> tuple[dict | None, int, set[str]]:
    """Resolve symbols in a single record's content.

    Returns:
        - resolved_content: dict with original + resolved fields, or None if no symbols
        - resolved_count: number of symbols resolved
        - unresolved: set of symbols encountered but not in lookup
    """
    resolved_content = content.copy()
    resolved_count = 0
    unresolved = set()

    for field, value in content.items():
        if value is None or not isinstance(value, str):
            continue

        value_str = str(value).strip()

        if value_str in symbol_lookup:
            # Known symbol - add resolved field
            resolved_content[f"{field}_resolved"] = symbol_lookup[value_str]
            resolved_count += 1
        elif _looks_like_symbol(value_str):
            # Looks like a symbol but not in dictionary
            unresolved.add(value_str)

    if resolved_count > 0:
        return resolved_content, resolved_count, unresolved
    return None, 0, unresolved


def _looks_like_symbol(value: str) -> bool:
    """Heuristic: short string (1-10 chars) that might be a symbol."""
    if len(value) < 1 or len(value) > 10:
        return False
    # Common symbol patterns
    SYMBOL_CHARS = set("○×△◎-*●□■◇◆")
    if any(c in SYMBOL_CHARS for c in value):
        return True
    # Short alphanumeric codes
    if value.isalnum() and len(value) <= 4:
        return True
    return False
```

### References

- [Source: docs/epics.md#Story-3.6] - Original story definition with acceptance criteria
- [Source: docs/architecture.md#Project-Structure] - Module location (line 68)
- [Source: docs/architecture.md#PostgreSQL-Schema] - extracted_records table schema
- [Source: docs/prd.md#Symbol-Resolution] - PRD feature description (FR11-13)
- [Source: docs/specs/extraction-pipeline-tech-spec.md] - Pipeline architecture, multi-pass extraction
- [Source: docs/sprint-artifacts/3-5-symbol-dictionary-detection.md] - Previous story patterns and learnings

## Dev Agent Record

### Context Reference

<!-- Path(s) to story context XML will be added here by context workflow -->

### Agent Model Used

Claude Opus 4.5 (claude-opus-4-5-20251101)

### Debug Log References

### Completion Notes List

- Implemented SymbolResolver class with full resolution workflow
- Added SymbolResolutionSummary dataclass to models.py
- Created 32 unit tests covering all acceptance criteria
- Added 9 integration tests to test_integration.py
- All 214 extraction tests pass
- Symbol heuristic detection uses SYMBOL_CHARS set for common symbols (○×△◎-*●□■◇◆✓✗☐☑)
- Batch size for database updates: 500 records per batch

### File List

**Created:**
- `src/extraction/symbol_resolver.py` - SymbolResolver class implementation

**Modified:**
- `src/extraction/models.py` - Added SymbolResolutionSummary dataclass
- `tests/extraction/test_integration.py` - Added symbol resolution integration tests

**Created:**
- `tests/extraction/test_symbol_resolver.py` - 32 unit tests for SymbolResolver