# Sprint Change Proposal

**Date:** 2025-11-27
**Author:** BMad Method - Correct Course Workflow
**Triggered by:** Story 3.5 (Symbol Dictionary Detection) implementation
**Scope:** Minor - Direct implementation adjustment

---

## Section 1: Issue Summary

**Problem Statement:**
Story 3.4a (Extraction Pipeline Orchestrator) only integrates Stories 3.1-3.4 for basic table extraction. The PRD and Architecture explicitly require **multi-pass extraction** that includes symbol dictionary detection (3.5), symbol resolution (3.6), and cross-reference detection (3.8). Without an integration story, the implemented symbol/cross-ref code will never execute in the pipeline.

**Discovery Context:**
During implementation of Story 3.5, it became clear that `src/extraction/pipeline.py` has no hooks to call `SymbolDetector` or `SymbolResolver`. Story 3.4a is already complete, so modifying it would be retroactive. A new integration story is needed.

**Evidence:**
- PRD Feature 3 explicitly describes 3-pass extraction (lines 117-152)
- Architecture ADR-003 mandates multi-pass pipeline
- Story 3.5 technical notes say "Run BEFORE content extraction"
- Sprint-status.yaml was missing Story 3.6 entirely

---

## Section 2: Impact Analysis

### Epic Impact

| Epic | Impact | Details |
|------|--------|---------|
| Epic 3 | **HIGH** | Stories 3.5, 3.6, 3.8 need pipeline integration |
| Epic 4 | **BLOCKED** | Story 4.1 depends on 3.6 (resolved content) |
| Epic 5 | **BLOCKED** | Depends on Epic 4 |

### Artifact Conflicts

| Artifact | Conflict? | Action Needed |
|----------|-----------|---------------|
| PRD | NO | None - PRD is correct |
| Architecture | NO | None - Architecture is correct |
| Epics Document | **YES** | Add new Story 3.8a for integration |
| Sprint Status | **YES** | Add missing Story 3.6, update statuses |

### Technical Impact

- `src/extraction/pipeline.py` needs to call symbol detection before table extraction
- `src/extraction/pipeline.py` needs to call symbol resolution after record building
- `src/extraction/pipeline.py` needs to call cross-reference detection after records exist

---

## Section 3: Recommended Approach

**Selected:** Direct Adjustment - Add Story 3.8a

**Rationale:**
- All feature stories (3.5, 3.6, 3.7, 3.8) should be implemented first
- Single integration story (3.8a) wires everything together at the end
- Avoids back-and-forth updates during feature development
- Clean "implement then integrate" pattern
- Story 3.4a remains unchanged (already done)

**Story Sequence:**
```
3.1  Table Boundary Detection      ✓ done
3.2  Multi-Level Header Extraction ✓ done
3.3  Merged Cell Resolution        ✓ done
3.4  Record Builder                ✓ done
3.4a Pipeline Orchestrator         ✓ done (basic flow)
3.5  Symbol Dictionary Detection   ✓ done
3.6  Symbol Resolution in Content  → implement
3.7  Custom Symbol Dictionary API  → implement
3.8  Cross-Reference Detection     → implement
3.8a Multi-Pass Pipeline Integration → WIRE UP ALL
```

**Effort:** Low
**Risk:** Low
**Timeline Impact:** Minimal - adds one story to Epic 3

---

## Section 4: Detailed Change Proposals

### Change 1: Add New Story 3.8a to Epics Document

**File:** `docs/epics.md`
**Location:** After Story 3.8, before Epic 4

```markdown
---

### Story 3.8a: Multi-Pass Pipeline Integration

As a **system**,
I want **the extraction pipeline to execute multi-pass extraction with symbol and cross-reference processing**,
So that **symbols are detected before content extraction and resolved before indexing**.

**Acceptance Criteria:**

**Given** an uploaded Excel document
**When** the extraction pipeline is triggered
**Then** the pipeline executes in the following order:

**PASS 1 - Symbol Dictionary Detection (before table extraction):**
1. Scans all sheets for symbol dictionary patterns using `SymbolDetector` (Story 3.5)
2. Stores detected symbols in `symbol_dictionaries` table
3. Builds in-memory symbol lookup for PASS 3

**PASS 2 - Table Extraction (existing 3.4a flow):**
4. Executes existing pipeline: TableDetector → HeaderExtractor → MergedCellResolver → RecordBuilder
5. Stores records in `extracted_records` table with `content` field

**PASS 3 - Symbol Resolution (after records exist):**
6. Resolves symbols in extracted content using `SymbolResolver` (Story 3.6)
7. Updates records with `resolved_content` JSONB field
8. Logs unresolved symbols as warnings

**PASS 4 - Cross-Reference Detection (after records exist):**
9. Detects cross-reference patterns using `CrossRefDetector` (Story 3.8)
10. Attempts to resolve references to target records
11. Stores in `cross_references` table with `resolved` flag

**And** progress tracking shows current pass:
- `detecting_symbols` stage for PASS 1
- `extracting_tables` stage for PASS 2
- `resolving_symbols` stage for PASS 3
- `detecting_references` stage for PASS 4

**And** extraction summary includes:
```python
{
    "symbols_detected": 15,
    "symbols_resolved": 142,
    "unresolved_symbols": 3,
    "cross_references_found": 45,
    "cross_references_resolved": 38
}
```

**Prerequisites:** Story 3.4a, 3.5, 3.6, 3.8

**Technical Notes:**
- Extend `ExtractionPipeline` class in `src/extraction/pipeline.py`
- Add `_run_symbol_detection()`, `_run_symbol_resolution()`, `_run_cross_ref_detection()` methods
- Modify `process_document()` to call passes in correct order
- Symbol detection MUST complete before table extraction begins
- Symbol resolution MUST have access to detected symbols from PASS 1
- Module: `src/extraction/pipeline.py`

**FR Coverage:** FR11-13 (Symbol resolution integration), FR15-17 (Cross-reference integration)
```

---

### Change 2: Update FR Coverage Map

**File:** `docs/epics.md`
**Section:** FR Coverage Map

**Update these rows to include 3.8a:**

| FR | Epic | Story |
|----|------|-------|
| FR11 | Epic 3 | 3.5, **3.8a** |
| FR12 | Epic 3 | 3.5, **3.8a** |
| FR13 | Epic 3 | 3.6, **3.8a** |
| FR15 | Epic 3 | 3.8, **3.8a** |
| FR16 | Epic 3 | 3.8, **3.8a** |
| FR17 | Epic 3 | 3.8, **3.8a** |

---

### Change 3: Update Epics Summary Table

**File:** `docs/epics.md`
**Section:** Epics Summary

**OLD:**
```
| 3 | Extraction Pipeline | 9 | FR6-17 |
```

**NEW:**
```
| 3 | Extraction Pipeline | 10 | FR6-17 |
```

**And update Total:**
```
**Total:** 5 epics, 32 stories
```

---

### Change 4: Update Sprint Status

**File:** `docs/sprint-artifacts/sprint-status.yaml`

**Changes:**
```yaml
  # Epic 3: Extraction Pipeline
  epic-3: contexted  # was: backlog
  3-1-table-boundary-detection: done  # was: review
  3-2-multi-level-header-extraction: done  # was: review
  3-3-merged-cell-resolution: done  # was: review
  3-4-record-builder: done  # was: review
  3-4a-extraction-pipeline-orchestrator: done  # was: review
  3-5-symbol-dictionary-detection: done  # was: in-progress
  3-6-symbol-resolution-in-content: backlog  # ADD - was missing
  3-7-custom-symbol-dictionary-upload: backlog
  3-8-cross-reference-detection-resolution: backlog
  3-8a-multi-pass-pipeline-integration: backlog  # ADD - new story
  epic-3-retrospective: optional
```

---

### Change 5: Update Story Prerequisites

**File:** `docs/epics.md`

**Story 4.1 Prerequisites - Update:**
```
**Prerequisites:** Story 3.8a (ensures multi-pass pipeline complete with symbol resolution)
```

**Rationale:** Story 4.1 (Structured Store) requires `resolved_content` which only exists after 3.8a integrates symbol resolution.

---

## Section 5: Implementation Handoff

**Change Scope:** Minor - Direct implementation by development team

### Deliverables

| Deliverable | Owner | Action |
|-------------|-------|--------|
| Add Story 3.8a to epics.md | Developer | Create new story section after 3.8 |
| Update sprint-status.yaml | Developer | Add 3.6, 3.8a; update statuses |
| Update FR Coverage Map | Developer | Add 3.8a to FR11-17 rows |
| Update Epics Summary | Developer | Epic 3: 10 stories, Total: 32 |
| Update Story 4.1 prerequisite | Developer | Change to 3.8a |

### Implementation Order

1. ✓ Story 3.5 (Symbol Detection) - **done**
2. → Story 3.6 (Symbol Resolution) - implement resolver component
3. → Story 3.7 (Custom Symbol API) - implement API endpoint
4. → Story 3.8 (Cross-Reference Detection) - implement detector component
5. → **Story 3.8a (Multi-Pass Integration)** - wire up 3.5, 3.6, 3.8 into pipeline
6. → Epic 4 can begin after 3.8a complete

### Success Criteria

1. Story 3.8a exists in epics.md with full acceptance criteria
2. Sprint-status.yaml includes Stories 3.6 and 3.8a
3. FR Coverage Map shows 3.8a for FR11-17
4. Story counts updated (Epic 3: 10, Total: 32)
5. After 3.8a implementation:
   - `ExtractionPipeline.process_document()` executes 4-pass flow
   - Symbol detection runs before table extraction
   - Symbol resolution runs after record building
   - Cross-reference detection runs after records exist
   - All Epic 3 tests pass

---

## Approval

- [ ] Proposal reviewed by: _______________
- [ ] Changes approved: Yes / No
- [ ] Implementation assigned to: _______________
- [ ] Target completion date: _______________

---

_Generated by BMad Method - Correct Course Workflow_
_Date: 2025-11-27_