# Word Document Support Epic - Planning Complete

**Date:** 2025-12-01
**Epic:** Word Document Processing Support
**Status:** Stories Ready for Development

---

## Summary

Successfully completed planning phase for Word document support in V2 extraction pipeline. Created comprehensive tech-spec, epic breakdown, and 4 detailed user stories ready for implementation.

---

## Deliverables Created

### 1. Technical Specification
**File:** `docs/feats/docx_support/tech-spec.md`

**Key Sections:**
- Problem statement: 51 Word documents (44% of CustomerDocument/) unprocessed
- Solution: Extend V2 pipeline with format normalization, preprocessing, and content extraction
- Implementation details with complete code examples
- Acceptance criteria: 51/51 files processed, ≥85% success rate

**Key Innovation:**
- Simpler than Excel: Single-mode markdown chunking (no dual-mode complexity)
- Docling HybridChunker with OpenAI tokenizer for document chunking
- Images/shapes/comments embedded inline before chunking
- LibreOffice integration for legacy .doc and .docm formats

### 2. Epic Document
**File:** `docs/feats/docx_support/epics.md`

**Epic Goal:** Process all 51 Word documents with ≥85% success rate

**Story Breakdown:**
- Story 1: Format Normalization & LibreOffice Integration (3 points)
- Story 2: Word Document Preprocessing (inline embedding) (5 points)
- Story 3: Markdown Conversion & Semantic Chunking (5 points)
- Story 4: Testing & Batch Processing Updates (3 points)

**Total Story Points:** 16

### 3. User Stories

#### Story 1: Format Normalization & LibreOffice Integration
**File:** `sprint_artifacts/story-word-support-1.md`
**Story Points:** 3
**Priority:** High

**Focus:**
- Implement .doc → .docx conversion via LibreOffice CLI
- Implement .docm → .docx conversion (macro stripping)
- Create DocumentToHtmlConverter (rename from ExcelToHtmlConverter)
- File validation and error handling

**Key Implementation:**
```python
def _convert_doc_to_docx(self, file_path: Path) -> Path:
    """Convert .doc or .docm to .docx using LibreOffice CLI."""
    subprocess.run([
        "libreoffice",
        "--headless",
        "--convert-to", "docx",
        "--outdir", str(temp_dir),
        str(file_path)
    ], check=True, timeout=60)
```

**Acceptance Criteria:**
- ✅ All 21 .doc files convert to .docx
- ✅ All 28 .docm files convert to .docx
- ✅ LibreOffice timeout and error handling
- ✅ File validation (size, ZIP structure)

#### Story 2: Word Document Preprocessing
**File:** `sprint_artifacts/story-word-support-2.md`
**Story Points:** 5
**Priority:** High

**Focus:**
- Extract images with AI-generated captions (GPT-4o vision)
- Extract shapes and text boxes
- Extract comments with ID markers
- Embed all inline in document text before Docling conversion
- Create WordPreprocessor class

**Key Components:**
- Image extraction + Azure OpenAI vision captioning
- Shape/annotation extraction
- Comment extraction with author/date
- Inline embedding: [IMAGE: ...], [COMMENT_1: ...], [SHAPE: ...]

**Acceptance Criteria:**
- ✅ All images extracted, captioned, and embedded inline
- ✅ Shapes and text boxes extracted and embedded inline
- ✅ Comments marked and embedded inline
- ✅ All 51 Word files preprocessed with inline content

#### Story 3: Markdown Conversion & Semantic Chunking
**File:** `sprint_artifacts/story-word-support-3.md`
**Story Points:** 5
**Priority:** High

**Focus:**
- Docling Word → Markdown conversion
- Docling HybridChunker with custom serializers
- Create SemanticChunker class
- Contextualize chunks for final text

**Key Features:**
- Single-mode extraction: Everything chunked using HybridChunker
- Tables preserved in markdown format within chunks
- OpenAI tokenizer (tiktoken) for text-embedding-3-large model
- 8000 token max per chunk
- Custom serializers for images and tables
- Chunk records with metadata (chunk_index, page_number, etc.)

**Acceptance Criteria:**
- ✅ Docling converts Word → Markdown
- ✅ Full markdown semantically chunked
- ✅ Tables preserved in markdown format within chunks
- ✅ Chunk records with complete metadata
- ✅ All 51 Word files processed

#### Story 4: Testing & Batch Processing Updates
**File:** `sprint_artifacts/story-word-support-4.md`
**Story Points:** 3
**Priority:** High

**Focus:**
- Update batch_extract_v2.py glob patterns (.doc/.docx/.docm)
- Comprehensive test suite for all 51 Word files
- Success rate validation (≥85%)
- Enhanced metrics (by_type: Excel/Word, by_format)
- Documentation updates

**Key Deliverables:**
- test_word_comprehensive.py (parametrized tests for 51 files)
- Enhanced status JSON with Word metrics
- Validation report generator
- Updated README and troubleshooting guide

**Acceptance Criteria:**
- ✅ All 51 Word files discovered and tracked
- ✅ Test suite validates success rate ≥85%
- ✅ Metrics by type and format
- ✅ Documentation updated

---

## Implementation Sequence

```
Story 1 (Format Normalization)
   ↓
Story 2 (Preprocessing)
   ↓
Story 3 (Content Extraction)
   ↓
Story 4 (Testing & Integration)
   ↓
Validation: 51 files processed, ≥85% success rate
```

**Dependencies:**
- Story 2 depends on Story 1 (needs .docx files)
- Story 3 depends on Story 2 (needs preprocessed documents)
- Story 4 depends on Story 3 (needs complete pipeline)

---

## Key Metrics to Track

**File Coverage:**
- Total Word files: 51
- By format: .docx: 2, .doc: 21, .docm: 28
- Currently processed: 0/51 (0%)
- Target: 51/51 (100%)

**Extraction Success:**
- Current: N/A (Word support not yet implemented)
- Target: ≥85% overall success rate (V2 + V1 fallback)
- Stretch goal: Match Excel pipeline (≥95%)

**Content Types Extracted:**
- Full document markdown (semantically chunked)
- Tables (preserved in markdown table format within chunks)
- Paragraphs, headings, lists (all in markdown)
- Images (with AI captions, embedded inline)
- Shapes/text boxes (embedded inline)
- Comments (with author/date, embedded inline)

---

## Technical Stack

**Runtime:**
- Python 3.11+
- FastAPI 0.122
- Celery 5.3+ with Redis 7.0+

**Document Processing:**
- Docling 2.63.0+ (Word → Document, HybridChunker)
- docling-core (HybridChunker, custom serializers)
- tiktoken (OpenAI tokenizer)
- python-docx (Word manipulation, already in uv.lock)
- LibreOffice 7.0+ CLI (.doc/.docm conversion)

**AI/ML:**
- Azure OpenAI GPT-4o (image captioning via vision API)

**Data Layer:**
- PostgreSQL 18 (SQLAlchemy 2.0+)
- Milvus 2.4+ (vector indexing)

**Testing:**
- pytest 8.0+ with pytest-asyncio 0.23+

---

## Files Created/Modified

### Modified Files
1. **src/extraction_v2/document_converter.py** - RENAME + MODIFY (from excel_to_html_converter.py)
   - Add Word format detection
   - Add .doc/.docm → .docx conversion
   - Excel: Docling → HTML, Word: Docling → Markdown

2. **src/extraction_v2/pipeline.py** - MODIFY
   - Add _extract_word() method
   - Route to Word or Excel pipeline
   - Integrate SemanticChunker for Word

3. **batch_extract_v2.py** - MODIFY
   - Add Word glob patterns (.docx/.doc/.docm)
   - Enhanced metrics (by_type, by_format)
   - Updated status JSON schema

### New Files Created
4. **src/extraction_v2/word_loader.py** - CREATE
   - Load Word documents with python-docx
   - Extract metadata
   - Validate file structure

5. **src/extraction_v2/word_preprocessor.py** - CREATE
   - Extract and caption images (GPT-4o vision)
   - Extract shapes/annotations
   - Extract comments
   - Embed all inline in document text

6. **src/extraction_v2/semantic_chunker.py** - CREATE
   - Docling HybridChunker with OpenAI tokenizer
   - Custom serializers (AnnotationPictureSerializer, MarkdownTableSerializer)
   - Contextualize chunks
   - Build chunk records with metadata

7. **tests/test_extraction_v2/test_word_conversion.py** - CREATE
   - Test .doc → .docx conversion
   - Test .docm → .docx conversion
   - Test LibreOffice integration

8. **tests/test_extraction_v2/test_word_preprocessing.py** - CREATE
   - Test image extraction and captioning
   - Test shape/comment extraction
   - Test inline embedding

9. **tests/test_extraction_v2/test_word_semantic_chunking.py** - CREATE
   - Test HybridChunker with OpenAI tokenizer
   - Test custom serializers
   - Test contextualization
   - Test chunk metadata generation

10. **tests/test_extraction_v2/test_word_comprehensive.py** - CREATE
    - End-to-end tests for all 51 Word files
    - Success rate validation

### Documentation Updates
11. **README.md** - MODIFY
    - Add Word document support section
    - Update batch processing examples
    - Document LibreOffice requirement

12. **docs/specs/extraction-pipeline-v2-tech-spec.md** - MODIFY
    - Add Word processing architecture
    - Update component specifications

---

## Next Steps

### Immediate Actions
1. **Start Story 1**: Format Normalization & LibreOffice Integration
   - Install LibreOffice (system dependency)
   - Implement conversion pipeline
   - Test with real .doc/.docm files

2. **Implement Stories 2-4 sequentially**:
   - Story 2: Word preprocessing (inline embedding)
   - Story 3: Markdown conversion & semantic chunking
   - Story 4: Testing and batch processing updates

3. **Run comprehensive validation**:
   - Execute test suite on all 51 files
   - Generate validation report
   - Verify ≥85% success rate

### Success Criteria Checklist
- [ ] 51/51 Word files discovered and tracked
- [ ] .doc and .docm files convert to .docx successfully
- [ ] Images/shapes/comments embedded inline in text
- [ ] Docling converts Word → Markdown (tables included)
- [ ] Full document chunked using Docling HybridChunker with OpenAI tokenizer
- [ ] ≥85% extraction success rate (V2 + V1 fallback)
- [ ] Automated test suite passing
- [ ] Documentation complete
- [ ] All code committed and reviewed

---

## Known Risks & Mitigation

**Risk 1: LibreOffice conversion failures**
- Mitigation: Timeout handling (60s)
- Mitigation: Retry logic
- Mitigation: Clear error messages
- Acceptance: Document conversion limitations

**Risk 2: Success rate below 85%**
- Mitigation: V1 fallback mechanism
- Mitigation: Error categorization
- Acceptance: Document known limitations for failed files

**Risk 3: Azure OpenAI rate limits (image captioning)**
- Mitigation: Batch images (up to 10 per request)
- Mitigation: Cache captions
- Mitigation: Graceful degradation if API fails

**Risk 4: .doc format complexity**
- Mitigation: LibreOffice handles legacy formats
- Mitigation: V1 fallback for problematic files
- Testing: Validate all 21 .doc files specifically

---

## Testing Strategy Summary

**Unit Tests:**
- Format conversion (LibreOffice integration)
- Image captioning (GPT-4o vision API)
- Content extraction (all types)
- Record building

**Integration Tests:**
- End-to-end pipeline for all 51 Word files
- Success rate validation
- Batch processor file discovery
- Metrics generation

**Validation:**
- Success rate ≥85%
- All 51 files processed
- Content types complete
- Error handling robust

---

## Documentation Updates

**README.md:**
- Word document support section
- LibreOffice installation instructions
- Batch processing examples with Word files
- Content types extracted

**Troubleshooting Guide:**
- LibreOffice conversion issues
- Image captioning failures
- .doc/.docm specific problems
- Success rate optimization tips

**Tech-Spec:**
- Complete Word processing architecture
- Preprocessing pipeline details
- Content extraction specifications

---

## Conclusion

Planning phase complete. All artifacts ready for implementation:
- ✅ Tech-spec created (comprehensive implementation guide)
- ✅ Epic defined (clear goal and success criteria)
- ✅ 4 user stories documented (sequential implementation)
- ✅ Implementation details specified (code examples included)
- ✅ Testing strategy defined (unit + integration + validation)
- ✅ Acceptance criteria clear (measurable outcomes)

**Ready to begin Story 1 implementation.**

---

**Planning Completed:** 2025-12-01
**Total Story Points:** 16
**Target Files:** 51 Word documents
**Target Success Rate:** ≥85%
**Feature Parity:** Full (matches Excel V2 pipeline)