# Story 3.7: Custom Symbol Dictionary Upload

Status: done

## Story

As an **API consumer**,
I want **to upload custom symbol dictionaries**,
So that **I can define meanings for domain-specific symbols**.

## Acceptance Criteria

1. **AC3.7.1:** Accept custom symbol upload via API
   - Endpoint: `POST /documents/{document_id}/symbols`
   - Request body:
     ```json
     {
       "symbols": [
         {"symbol": "★", "meaning": "Priority item"},
         {"symbol": "△", "meaning": "Pending review"}
       ]
     }
     ```
   - Validate document_id exists and belongs to authenticated user
   - Return 404 if document not found
   - Return 400 if symbols array is empty or malformed

2. **AC3.7.2:** Store custom symbols in database
   - Store in `symbol_dictionaries` table with document_id
   - Mark symbols as `is_custom: true` to distinguish from auto-detected
   - Include source as "api_upload" in context field
   - Return 200 OK with response:
     ```json
     {
       "document_id": "uuid",
       "symbols_added": 2,
       "message": "Custom symbols will be applied to this document"
     }
     ```

3. **AC3.7.3:** Custom symbols take precedence over auto-detected
   - When resolving symbols, check custom symbols first
   - Custom symbols override auto-detected symbols with same key
   - Log when custom symbol overrides auto-detected (info level)

4. **AC3.7.4:** Re-trigger symbol resolution if document already processed
   - Check if document status is "completed" (already processed)
   - If already processed, queue Celery task to re-run symbol resolution
   - Update document status to "reprocessing" during re-resolution
   - Return to "completed" when re-resolution finishes

5. **AC3.7.5:** Validate symbol input
   - Symbol string must be 1-50 characters
   - Meaning string must be 1-500 characters
   - Maximum 100 symbols per request
   - Return validation errors with specific field information

6. **AC3.7.6:** Support symbol deletion
   - Endpoint: `DELETE /documents/{document_id}/symbols/{symbol}`
   - Remove custom symbol from database
   - Re-trigger resolution if document already processed
   - Return 404 if symbol not found

## Tasks / Subtasks

- [x] **Task 1: Create API schema models** (AC: 3.7.1, 3.7.5)
  - [x] Create `SymbolUploadRequest` Pydantic model in `src/api/schemas/symbols.py`
  - [x] Create `SymbolEntry` model with symbol, meaning fields
  - [x] Create `SymbolUploadResponse` model
  - [x] Add field validation (length constraints)

- [x] **Task 2: Implement POST endpoint** (AC: 3.7.1, 3.7.2)
  - [x] Add route `POST /documents/{document_id}/symbols` to `src/api/routes/documents.py`
  - [x] Verify document exists and belongs to API key
  - [x] Call service layer to store symbols
  - [x] Return appropriate response

- [x] **Task 3: Create symbol upload service** (AC: 3.7.2, 3.7.3)
  - [x] Create `src/services/symbol_service.py`
  - [x] Implement `upload_custom_symbols(document_id, symbols, session) -> SymbolUploadResult`
  - [x] Mark custom symbols with `context="api_upload"` or add `is_custom` flag
  - [x] Handle duplicate symbols (update meaning if exists)

- [x] **Task 4: Modify SymbolResolver for custom precedence** (AC: 3.7.3)
  - [x] Update `_build_symbol_lookup()` in `src/extraction/symbol_resolver.py`
  - [x] Load custom symbols last to ensure precedence
  - [x] Log when custom symbol overrides auto-detected

- [x] **Task 5: Implement re-resolution trigger** (AC: 3.7.4)
  - [x] Check document status in upload endpoint
  - [x] Create Celery task `re_resolve_symbols` in `src/workers/tasks.py`
  - [x] Queue task if document status is "completed"
  - [x] Update document status during re-processing

- [x] **Task 6: Implement DELETE endpoint** (AC: 3.7.6)
  - [x] Add route `DELETE /documents/{document_id}/symbols/{symbol}` to documents.py
  - [x] Implement symbol deletion in service layer
  - [x] Trigger re-resolution if needed
  - [x] Return appropriate response

- [x] **Task 7: Unit tests** (AC: All)
  - [x] Create `tests/api/test_symbol_routes.py`
  - [x] Test POST endpoint success case
  - [x] Test POST endpoint validation errors
  - [x] Test DELETE endpoint
  - [x] Test 404 for non-existent document
  - [x] Test custom symbol precedence

- [x] **Task 8: Integration tests** (AC: All)
  - [x] Test full flow: upload document → upload custom symbols → verify resolution
  - [x] Test re-resolution after custom symbol upload
  - [x] Test custom symbols override auto-detected

## Dev Notes

### Architecture Patterns and Constraints

From architecture.md:
- **Route location:** `src/api/routes/documents.py` (architecture.md:51)
- **Service location:** `src/services/symbol_service.py` (new file)
- **Naming conventions:**
  - Route: snake_case function names (`upload_custom_symbols`)
  - Schema: PascalCase classes (`SymbolUploadRequest`)
  - Service: snake_case functions
- **Error handling:** Return proper HTTP status codes with DSOLError subclasses
- **Authentication:** Use existing API key auth dependency

From epics.md Story 3.7:
- **Prerequisites:** Story 3.6 (Symbol Resolution in Content)
- **FR Covered:** FR14 (Users can upload custom symbol dictionaries)
- **Celery task:** Re-resolve if document already processed

### Learnings from Previous Story

**From Story 3.6: Symbol Resolution in Content (Status: done)**

- **SymbolResolver available at:** `src/extraction/symbol_resolver.py`
  - Use `SymbolResolver.resolve_document(document_id, session)` to resolve symbols
  - Returns `SymbolResolutionSummary` with resolution statistics
  - Batch updates resolved_content in extracted_records table

- **Data Models:**
  - `SymbolResolutionSummary` dataclass in `src/extraction/models.py`
  - All dataclasses have `to_dict()` methods

- **Database Pattern:**
  - ORM model `SymbolDictionary` in `src/db/models.py`
  - Symbol lookup: `session.query(SymbolDictionary).filter_by(document_id=doc_id)`
  - `_build_symbol_lookup()` returns `{symbol: meaning}` dictionary

- **Component Pattern:**
  - Use structlog with context parameters
  - Handle errors gracefully, log warnings for non-critical issues

- **Testing Pattern:**
  - Unit tests in `tests/extraction/test_symbol_resolver.py`
  - Integration tests in `tests/extraction/test_integration.py`
  - Test both positive and negative cases

[Source: docs/sprint-artifacts/3-6-symbol-resolution-in-content.md#Dev-Notes]

### Project Structure Notes

Expected module structure after Story 3.7:
```
src/api/
├── routes/
│   ├── documents.py         # MODIFY: Add symbol upload endpoints
│   └── ...
├── schemas/
│   ├── symbols.py           # NEW: Symbol upload schemas
│   └── ...

src/services/
├── symbol_service.py        # NEW: Symbol upload service

src/extraction/
├── symbol_resolver.py       # MODIFY: Custom symbol precedence

src/workers/
├── tasks.py                 # MODIFY: Add re_resolve_symbols task

tests/
├── api/
│   └── test_symbol_routes.py  # NEW: Symbol endpoint tests
```

### Database Schema Reference

From Story 1.2 (symbol_dictionaries):
```sql
CREATE TABLE symbol_dictionaries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    sheet_name VARCHAR(255),
    symbol VARCHAR(50) NOT NULL,
    meaning TEXT NOT NULL,
    context VARCHAR(255),           -- Use "api_upload" for custom symbols
    created_at TIMESTAMP DEFAULT NOW()
);
```

Note: The `context` field can be used to distinguish custom symbols (`context="api_upload"`) from auto-detected ones (`context=null` or sheet name).

### API Design

**POST /documents/{document_id}/symbols**
```python
@router.post("/{document_id}/symbols", response_model=SymbolUploadResponse)
async def upload_custom_symbols(
    document_id: UUID,
    request: SymbolUploadRequest,
    api_key: APIKey = Depends(get_api_key),
    session: Session = Depends(get_session),
) -> SymbolUploadResponse:
    """Upload custom symbol definitions for a document."""
    # 1. Verify document exists and belongs to API key
    # 2. Store symbols via service
    # 3. Trigger re-resolution if needed
    # 4. Return response
```

**DELETE /documents/{document_id}/symbols/{symbol}**
```python
@router.delete("/{document_id}/symbols/{symbol}")
async def delete_custom_symbol(
    document_id: UUID,
    symbol: str,
    api_key: APIKey = Depends(get_api_key),
    session: Session = Depends(get_session),
) -> dict:
    """Delete a custom symbol from a document."""
```

### References

- [Source: docs/epics.md#Story-3.7] - Original story definition with acceptance criteria
- [Source: docs/architecture.md#Project-Structure] - Route and service locations
- [Source: docs/prd.md#FR14] - Users can upload custom symbol dictionaries
- [Source: docs/sprint-artifacts/3-6-symbol-resolution-in-content.md] - Previous story patterns

## Dev Agent Record

### Context Reference

<!-- Path(s) to story context XML will be added here by context workflow -->

### Agent Model Used

Claude Opus 4.5

### Debug Log References

N/A - No debug logs needed.

### Completion Notes List

1. Created Pydantic schemas for symbol upload API with comprehensive validation:
   - `SymbolEntry`: Individual symbol with 1-50 char symbol, 1-500 char meaning
   - `SymbolUploadRequest`: Array of 1-100 unique symbols
   - `SymbolUploadResponse`, `SymbolDeleteResponse`, `SymbolListResponse`

2. Implemented async SymbolService with:
   - `upload_custom_symbols()` - adds/updates symbols with `context="api_upload"`
   - `delete_custom_symbol()` - removes custom symbols only
   - `get_document_symbols()` - lists all document symbols
   - `check_document_needs_reprocessing()` - checks if document is "completed"

3. Added three new API endpoints to documents router:
   - `POST /{document_id}/symbols` - upload custom symbols
   - `DELETE /{document_id}/symbols/{symbol}` - delete custom symbol
   - `GET /{document_id}/symbols` - list all document symbols

4. Modified SymbolResolver to prioritize custom symbols:
   - Order query to load auto-detected symbols first, custom last
   - Custom symbols (context="api_upload") override auto-detected
   - Log when override occurs

5. Created Celery task `re_resolve_symbols` to:
   - Update document status to "reprocessing"
   - Re-run SymbolResolver.resolve_document()
   - Return status to "completed"

6. Tests: 52 tests passing (33 resolver + 14 API + 5 integration)

### File List

**Created:**
- `src/api/schemas/symbols.py` - Pydantic schemas for symbol API
- `src/services/symbol_service.py` - Symbol upload service layer
- `tests/api/test_symbol_routes.py` - Unit tests for symbol API
- `tests/integration/test_symbol_integration.py` - Integration tests

**Modified:**
- `src/api/schemas/__init__.py` - Export new symbol schemas
- `src/api/routes/documents.py` - Added symbol upload/delete/list endpoints
- `src/extraction/symbol_resolver.py` - Custom symbol precedence in lookup
- `src/workers/tasks.py` - Added re_resolve_symbols Celery task
- `tests/extraction/test_symbol_resolver.py` - Updated mocks for order_by chain