# Epic: Word Document Processing Support

**Epic ID:** word-document-support
**Project:** DSOL
**Created:** 2025-12-01
**Status:** Ready for Development

---

## Epic Goal

Extend the V2 extraction pipeline to process Word documents (.doc, .docx, .docm) with full feature parity to Excel processing, achieving ≥85% extraction success rate across all 51 Word files in CustomerDocument/.

---

## Epic Scope

**In Scope:**
- Format normalization for legacy .doc and macro-enabled .docm files
- Word document preprocessing (images, shapes, comments with inline embedding)
- Docling-based Word → Markdown conversion
- Docling HybridChunker with OpenAI tokenizer (tiktoken) for chunking
- Batch processing updates for Word file discovery
- Comprehensive testing suite for all 51 Word files
- Documentation and troubleshooting guides

**Out of Scope:**
- PowerPoint (.pptx) processing
- PDF processing
- Word document editing/modification
- OCR for scanned documents
- Macro execution
- Advanced formatting preservation

---

## Success Criteria

1. ✅ All 51 Word documents discovered and tracked (21 .doc, 2 .docx, 28 .docm)
2. ✅ Format normalization working (.doc/.docm → .docx via LibreOffice)
3. ✅ Images extracted with AI-generated captions (embedded inline)
4. ✅ Shapes, annotations, and comments extracted (embedded inline)
5. ✅ Docling converts Word → Document with full structure preserved
6. ✅ Document chunked using Docling HybridChunker with OpenAI tokenizer
7. ✅ ≥85% extraction success rate (V2 + V1 fallback combined)
8. ✅ Batch processor tracks Word files with metrics
9. ✅ Automated test suite validates all 51 files
10. ✅ Documentation updated with Word support

---

## Story Breakdown

### Story 1: Format Normalization & LibreOffice Integration
**File:** `story-word-support-1.md`
**Focus:** .doc/.docm → .docx conversion pipeline

**Key Tasks:**
- Implement LibreOffice CLI integration
- Add .doc → .docx conversion
- Add .docm → .docx conversion (macro stripping)
- File validation and error handling
- Timeout and retry logic

**Acceptance:** All .doc and .docm files successfully converted to .docx

---

### Story 2: Word Document Preprocessing
**File:** `story-word-support-2.md`
**Focus:** Extract and embed images, shapes, comments inline

**Key Tasks:**
- Extract images with context
- Generate AI captions via Azure OpenAI GPT-4o vision
- Extract shapes and annotations
- Extract comments with ID markers
- Embed all inline in document text
- Create WordPreprocessor class

**Acceptance:** All Word documents preprocessed with images/shapes/comments embedded inline in text

---

### Story 3: Document Chunking with HybridChunker
**File:** `story-word-support-3.md`
**Focus:** Docling HybridChunker with OpenAI tokenizer and custom serializers

**Key Tasks:**
- Integrate Docling document conversion
- Create SemanticChunker using HybridChunker with OpenAITokenizer
- Configure custom serializers (AnnotationPictureSerializer, MarkdownTableSerializer)
- Implement contextualization for chunks
- Build chunk records with metadata
- Update ExtractionPipelineV2.extract_from_file()

**Acceptance:** Document chunked using HybridChunker, tables and images preserved within chunks

---

### Story 4: Testing & Batch Processing Updates
**File:** `story-word-support-4.md`
**Focus:** Comprehensive testing and batch processor updates

**Key Tasks:**
- Update batch_extract_v2.py glob patterns (.doc/.docx/.docm)
- Comprehensive test suite for all 51 Word files
- Success rate validation (≥85%)
- Error categorization for Word-specific issues
- Metrics tracking by format
- Update documentation (README, troubleshooting)

**Acceptance:** Test suite passes, 51 files tracked, ≥85% success rate, docs updated

---

## Implementation Sequence

```
Story 1 (Format Normalization)
   ↓
Story 2 (Preprocessing) ← Can start in parallel once Story 1 basics complete
   ↓
Story 3 (Content Extraction)
   ↓
Story 4 (Testing & Integration)
   ↓
Validation: Run full test suite, verify 85% success rate
```

**Dependencies:**
- Story 2 depends on Story 1 (needs .docx files)
- Story 3 depends on Story 2 (needs preprocessed documents)
- Story 4 depends on Story 3 (needs complete pipeline)

---

## Technical Context

**From Tech-Spec:**
- 51 Word documents total (.doc: 21, .docx: 2, .docm: 28)
- Currently 0/51 files processed (44% of CustomerDocument/)
- Target: ≥85% extraction success rate
- Reuse V2 pipeline architecture from Excel

**Stack:**
- Python 3.11+, FastAPI 0.122, Celery 5.3+
- Docling 2.63.0+ (Word → Document, HybridChunker)
- docling-core (HybridChunker, custom serializers)
- tiktoken (OpenAI tokenizer)
- python-docx (Word manipulation)
- LibreOffice 7.0+ CLI (format conversion)
- Azure OpenAI GPT-4o (image captioning)
- PostgreSQL 18, Milvus 2.4+

**Key Files:**
- `src/extraction_v2/document_converter.py` (create/modify)
- `src/extraction_v2/word_loader.py` (create)
- `src/extraction_v2/word_preprocessor.py` (create)
- `src/extraction_v2/semantic_chunker.py` (create)
- `src/extraction_v2/pipeline.py` (modify)
- `batch_extract_v2.py` (modify)

---

## Epic Value

**Business Impact:**
- Process 44% more documents in CustomerDocument/ folder
- Complete document coverage for production readiness
- Competitive parity with full-document processing systems
- Extract valuable structured data from Word documents

**Developer Impact:**
- Reusable preprocessing patterns
- Unified pipeline for multiple document types
- Comprehensive test coverage
- Clear troubleshooting guides

---

## Epic Notes

- Leverages existing V2 pipeline architecture from Excel
- LibreOffice CLI pattern follows .xls/.xlsm conversion approach
- Simpler than Excel: Single-mode chunking with HybridChunker (no dual-mode)
- Tables preserved in markdown format within chunks (no separate extraction)
- Docling HybridChunker with OpenAI tokenizer for accurate token counting
- Custom serializers for images and tables
- Focus on content extraction, not formatting preservation

---

**Epic Status:** Ready for Sprint Planning