# DSOL - Technical Specification: Word Document Processing Support **Author:** TextIQ **Date:** 2025-12-01 **Project Level:** Feature Enhancement **Change Type:** Document Format Support Extension **Development Context:** Brownfield - Extending V2 Extraction Pipeline --- ## Context ### Available Documents **Loaded Documentation:** - ✓ Extraction Pipeline V2 Tech-Spec (docs/specs/extraction-pipeline-v2-tech-spec.md) - Comprehensive 1,232-line specification for Excel processing - ✓ V2 Pipeline Testing Epic (docs/feats/wrapup_pipeline_v2/) - Recent testing and enhancement work - ✓ Product Brief (docs/brief.md) - ✓ Project Configuration (pyproject.toml, .bmad/bmm/config.yaml) **Current State:** - V2 extraction pipeline fully operational for Excel files (.xlsx, .xls, .xlsm) - Batch processing script (`batch_extract_v2.py`) handles 58 Excel files - 51 Word documents in CustomerDocument/ folder **currently unprocessed** - python-docx library already in dependencies (uv.lock) but not actively used - Docling 2.63.0+ supports both Excel and Word document conversion ### Project Stack **Runtime Environment:** - **Language:** Python 3.11+ - **Framework:** FastAPI 0.122 - **API Server:** Uvicorn with async support - **Task Queue:** Celery 5.3+ with Redis 7.0+ **Data Layer:** - **Database:** PostgreSQL 18 (SQLAlchemy 2.0+, asyncpg, alembic) - **Vector Store:** Milvus 2.4+ (pymilvus) **Document Processing:** - **Document Conversion:** Docling 2.63.0+ (Excel → HTML, **Word → Markdown**) - **Excel Parsing:** openpyxl 3.1.5, xlrd 2.0.2+ (.xls support) - **Word Document Parsing:** python-docx (already available) - **Legacy Format Conversion:** LibreOffice 7.0+ CLI (.doc/.docm → .docx) - **Document Chunking:** Docling HybridChunker with OpenAI tokenizer (tiktoken) - **LLM Integration:** Azure OpenAI (GPT-4o) via openai 1.0+ **Development Tools:** - **Testing:** pytest 8.0+, pytest-asyncio 0.23+ - **Code Quality:** black 24.0+, ruff 0.5+, mypy 1.10+ - **Logging:** structlog 24.0+ ### Existing Codebase Structure **Project Organization:** ``` dsol/ ├── src/ │ ├── extraction_v2/ # V2 pipeline (currently Excel-only) │ │ ├── pipeline.py # Main orchestrator - MODIFY │ │ ├── excel_to_html_converter.py # Rename/extend to document_to_html_converter.py │ │ ├── html_table_extractor.py # Reuse as-is │ │ ├── llm_header_detector.py # Reuse as-is │ │ ├── record_builder.py # Reuse as-is │ │ ├── detect_border.py # Reuse as-is (tables in Word docs) │ │ └── excel_loader.py # Create word_loader.py equivalent │ ├── workers/ # Celery tasks │ │ └── extraction_v2_tasks.py - MODIFY │ ├── api/ # FastAPI routes │ └── db/ # Database models ├── tests/ # Test suite │ └── test_extraction_v2/ # V2 pipeline tests - EXTEND ├── batch_extract_v2.py # Batch processing - MODIFY └── CustomerDocument/ # Test data ├── [58 Excel files] # Currently processed └── [51 Word files] # Target for this work ├── 21 .doc files ├── 2 .docx files └── 28 .docm files ``` **Key Existing Modules:** - `ExtractionPipelineV2` - Main pipeline orchestrator (extend for Word support) - `ExcelToHtmlConverter` - Handles .xlsx/.xls/.xlsm conversion (pattern to replicate) - `HtmlTableExtractor` - Parses Docling HTML output (reuse unchanged) - `LlmHeaderDetector` - GPT-4o header detection (reuse unchanged) - `RecordBuilder` - Generates JSON records (reuse unchanged) - `batch_extract_v2.py` - Batch processing with status tracking (extend patterns) --- ## The Change ### Problem Statement **Current Situation:** The V2 extraction pipeline successfully processes Excel files but cannot handle Word documents, leaving 51 files (44% of CustomerDocument/) unprocessed. **Specific Issues:** 1. **Unsupported File Formats:** V2 pipeline only recognizes Excel formats - 51 Word documents unprocessed (.doc: 21, .docx: 2, .docm: 28) - No Word file patterns in batch processor glob - Pipeline rejects Word files at format detection stage 2. **Missing Format Normalization:** No .doc/.docm conversion pipeline - Legacy .doc format not supported by Docling directly - Macro-enabled .docm files need conversion - No LibreOffice integration for Word formats 3. **No Word-Specific Preprocessing:** Excel has rich preprocessing, Word needs equivalent - Images/shapes extraction and AI captioning - Comments extraction and markers - Table border detection (Word tables) - Paragraph structure preservation 4. **Incomplete Content Extraction:** Current focus is tables only - Need to extract non-tabular content (paragraphs, lists, headings) - Images and shapes should be captioned and embedded - Comments should be preserved with markers **Business Impact:** - Cannot process 44% of CustomerDocument/ files - Incomplete document coverage limits system usefulness - Word documents contain valuable structured data (tables, forms) - Competitive gap - full-document processing needed for production ### Proposed Solution **Solution Overview:** Extend V2 extraction pipeline to support Word documents (.doc, .docx, .docm) with full feature parity to Excel processing, including tabular and non-tabular content extraction. **Four-Part Approach:** **1. Format Normalization Pipeline (Legacy Support)** - Implement .doc → .docx conversion using LibreOffice CLI - Implement .docm → .docx conversion using LibreOffice CLI - Follow existing pattern from Excel (.xls → .xlsx, .xlsm → .xlsx) - Validate converted files before Docling processing **2. Word Document Preprocessing** - Extract images with AI-generated captions (Azure OpenAI vision) - Extract shapes and annotations with descriptions - Extract comments and apply ID markers - Detect table borders for accurate table identification - Preserve paragraph structure and formatting **3. Docling Word → Markdown Conversion** - Leverage Docling's native Word support - Convert .docx to Markdown (tables, text, images all in markdown format) - Preserve document structure (headings, lists, tables) - No separate table extraction needed **4. Docling HybridChunker with Custom Serializers** - Chunk the entire markdown output (tables included) - **Chunking Strategy:** Recursive character splitter with markdown-aware separators - Tables preserved in markdown table format within chunks - Unified record building with chunk metadata and source traceability **Expected Outcome:** - ✅ All 51 Word documents processed through V2 pipeline - ✅ Markdown conversion via Docling (tables, text, images preserved) - ✅ Full document semantically chunked using Docling - ✅ Tables preserved in markdown format within chunks - ✅ ≥85% extraction success rate (V2 + V1 fallback) - ✅ Context-preserving retrieval via semantic chunking ### Scope **In Scope:** 1. **Word Format Support** - Add .doc, .docx, .docm support to V2 pipeline - Format normalization: .doc → .docx, .docm → .docx (LibreOffice CLI) - File validation and error handling 2. **Document Preprocessing** - Image extraction with AI captioning - Shape/annotation extraction and embedding - Comment extraction with ID markers - Text content enrichment before chunking 3. **Markdown Conversion & Semantic Chunking** - Docling Word → Markdown conversion - Docling semantic chunking of full markdown (tables included) - **Chunking Configuration:** 1000-character chunks with 200-character overlap - Unified JSON record output with chunk metadata - Source traceability for all chunks 4. **Batch Processing Updates** - Update glob patterns to include .doc/.docx/.docm - Enhanced status tracking for Word files - Error categorization (Word-specific) - Metrics reporting 5. **Testing & Validation** - Unit tests for Word conversion and preprocessing - Integration tests for all 51 Word files - Success rate validation (≥85% target) - Comparison with Excel pipeline quality 6. **Documentation** - Update README with Word support notes - Document preprocessing pipeline for Word - Troubleshooting guide for common Word issues **Out of Scope:** 1. ❌ PowerPoint (.pptx) processing (future enhancement) 2. ❌ PDF processing (separate pipeline) 3. ❌ Word document editing or modification 4. ❌ OCR for scanned/image-based documents 5. ❌ Advanced Word formatting preservation (focus on content) 6. ❌ Mail merge or template processing 7. ❌ Macro execution (.docm macros not run) --- ## Implementation Details ### Source Tree Changes **File Modifications:** 1. **src/extraction_v2/document_converter.py** - CREATE (generalize from excel_to_html_converter.py) - DocumentConverter class for both Excel and Word - Excel: Docling → HTML (existing logic) - Word: Docling → Markdown (new logic) - Add Word format detection - Add .doc/.docm → .docx conversion via LibreOffice - Add Word-specific preprocessing 2. **src/extraction_v2/word_loader.py** - CREATE - Load Word documents with python-docx - Extract metadata (similar to excel_loader.py) - Validate Word file structure - Provide document object for preprocessing 3. **src/extraction_v2/word_preprocessor.py** - CREATE - Extract images from Word docs - Generate AI captions for images (Azure OpenAI vision) - Extract shapes and annotations - Extract comments with ID markers - Detect table boundaries - Preserve paragraph structure 4. **src/extraction_v2/semantic_chunker.py** - CREATE - Docling-based semantic chunking for Word documents - RecursiveCharacterTextSplitter with semantic separators - Chunk size: 1000 characters, overlap: 200 characters - Build chunk records with metadata (chunk_index, total_chunks, etc.) 5. **src/extraction_v2/pipeline.py** - MODIFY - Update `extract_from_file()` to detect Word formats - Add Word processing branch parallel to Excel branch - Word: Markdown conversion → Semantic chunking - Excel: HTML conversion → Table extraction (unchanged) 6. **batch_extract_v2.py** - MODIFY - Line ~222: Add Word patterns to glob search - Update file validation for Word formats - Enhanced error categorization for Word-specific issues - Metrics tracking by format (.doc/.docx/.docm) **New Test Files:** 7. **tests/test_extraction_v2/test_word_conversion.py** - CREATE - Test .doc → .docx conversion - Test .docm → .docx conversion - Test Docling Word → HTML conversion - Validate HTML structure 8. **tests/test_extraction_v2/test_word_preprocessing.py** - CREATE - Test image extraction and captioning - Test shape/annotation extraction - Test comment extraction - Test table border detection 9. **tests/test_extraction_v2/test_word_semantic_chunking.py** - CREATE - Test Docling chunker initialization - Test chunk size and overlap configuration - Test semantic boundary detection - Test chunk metadata generation 10. **tests/test_extraction_v2/test_word_comprehensive.py** - CREATE - End-to-end tests for all 51 Word files - Success rate validation - Quality comparison with Excel pipeline **Documentation Updates:** 11. **README.md** - MODIFY - Add Word document support section - Document .doc/.docm conversion requirements - Update batch processing examples 12. **docs/specs/extraction-pipeline-v2-tech-spec.md** - MODIFY - Add Word processing architecture - Update component specifications - Document preprocessing pipeline for Word ### Technical Approach **1. Format Normalization (LibreOffice CLI)** Extend existing Excel pattern to Word formats: ```python # In document_to_html_converter.py def _convert_doc_to_docx(self, file_path: Path) -> Path: """Convert .doc to .docx using LibreOffice CLI.""" if not shutil.which("libreoffice"): raise RuntimeError("LibreOffice not installed") temp_dir = Path(tempfile.mkdtemp()) output_path = temp_dir / f"{file_path.stem}.docx" try: subprocess.run( [ "libreoffice", "--headless", "--convert-to", "docx", "--outdir", str(temp_dir), str(file_path) ], check=True, capture_output=True, timeout=60 ) converted_file = temp_dir / f"{file_path.stem}.docx" if not converted_file.exists(): raise RuntimeError(f"Conversion failed: {file_path.name}") return converted_file except subprocess.TimeoutExpired: raise RuntimeError(f"Conversion timeout: {file_path.name}") ``` **2. Word Document Preprocessing** ```python # In word_preprocessor.py class WordPreprocessor: """Preprocess Word documents before Docling conversion.""" def __init__(self, openai_client): self.openai_client = openai_client self.logger = structlog.get_logger() def preprocess(self, file_path: Path) -> Path: """ Preprocess Word document: 1. Extract images with AI captions 2. Extract shapes/annotations 3. Extract comments 4. Detect table borders 5. Preserve paragraph structure Returns path to preprocessed .docx file. """ doc = Document(file_path) # Extract and caption images images = self._extract_images(doc) image_captions = self._generate_image_captions(images) self._embed_image_captions(doc, image_captions) # Extract shapes and annotations shapes = self._extract_shapes(doc) self._embed_shape_descriptions(doc, shapes) # Extract and mark comments comments = self._extract_comments(doc) self._apply_comment_markers(doc, comments) # Detect table borders (similar to Excel) self._detect_table_boundaries(doc) # Save preprocessed document output_path = file_path.parent / f"{file_path.stem}_preprocessed.docx" doc.save(str(output_path)) return output_path def _extract_images(self, doc: Document) -> List[Dict]: """Extract all images from document with context.""" images = [] for rel_id, rel in doc.part.rels.items(): if "image" in rel.target_ref: img_data = rel.target_part.blob # Extract image context (surrounding paragraphs) context = self._get_image_context(doc, rel_id) images.append({ "rel_id": rel_id, "data": img_data, "context": context }) return images def _generate_image_captions(self, images: List[Dict]) -> List[str]: """Generate AI captions for images using Azure OpenAI vision.""" captions = [] for img in images: # Use Azure OpenAI GPT-4o with vision caption = self.openai_client.chat.completions.create( model="gpt-4o", messages=[ { "role": "user", "content": [ {"type": "text", "text": f"Describe this image in context: {img['context']}"}, {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64.b64encode(img['data']).decode()}"}} ] } ] ) captions.append(caption.choices[0].message.content) return captions def _extract_comments(self, doc: Document) -> List[Dict]: """Extract all comments with author and text.""" comments = [] for comment in doc.part.comments_part.comments: comments.append({ "id": comment.id, "author": comment.author, "text": comment.text, "range": comment.range }) return comments ``` **3. Docling Word → Markdown Conversion** ```python # In document_converter.py def convert_word_to_markdown(self, file_path: str) -> Optional[str]: """ Convert Word document to Markdown using Docling. Pipeline: 1. Convert .doc/.docm → .docx if needed 2. Preprocess with WordPreprocessor (images/comments/shapes) 3. Docling conversion to Markdown 4. Return Markdown string (tables, text, headings all in markdown) """ file_path = Path(file_path) # Step 1: Format normalization if file_path.suffix in [".doc", ".docm"]: file_path = self._convert_doc_to_docx(file_path) # Step 2: Preprocessing preprocessor = WordPreprocessor(self.openai_client) preprocessed_path = preprocessor.preprocess(file_path) # Step 3: Docling conversion to Markdown result = self.converter.convert(str(preprocessed_path)) if result and result.document: markdown = result.document.export_to_markdown() self.logger.info("Word → Markdown conversion successful", file=file_path.name) return markdown else: self.logger.error("Word → Markdown conversion failed", file=file_path.name) return None ``` **4. Semantic Chunking with Docling** ```python # In semantic_chunker.py import tiktoken from docling_core.transforms.chunker.hybrid_chunker import HybridChunker from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer from docling_core.transforms.chunker.hierarchical_chunker import ( ChunkingDocSerializer, ChunkingSerializerProvider, ) from docling_core.transforms.serializer.markdown import ( MarkdownTableSerializer, MarkdownPictureSerializer, ) from docling_core.types.doc.document import ( PictureClassificationData, PictureDescriptionData, PictureItem, DoclingDocument, ) from docling_core.transforms.serializer.base import BaseDocSerializer, SerializationResult from docling_core.transforms.serializer.common import create_ser_result from typing import List, Dict, Any import structlog class AnnotationPictureSerializer(MarkdownPictureSerializer): """Custom serializer for picture annotations that includes captions and descriptions.""" def serialize( self, *, item: PictureItem, doc_serializer: BaseDocSerializer, doc: DoclingDocument, **kwargs: Any, ) -> SerializationResult: """Serialize picture item with captions and annotations.""" text_parts: list[str] = [] # Add caption if available caption = item.caption_text(doc=doc) if caption: text_parts.append(f"[CAPTION]: {caption}") # Add annotations for annotation in item.annotations: if isinstance(annotation, PictureClassificationData): predicted_class = ( annotation.predicted_classes[0].class_name if annotation.predicted_classes else None ) if predicted_class is not None: text_parts.append(f"[PICTURE TYPE]: {predicted_class}") elif isinstance(annotation, PictureDescriptionData): text_parts.append(f"[PICTURE DESCRIPTION] {annotation.text} [END OF PICTURE DESCRIPTION]") text_res = "\n".join(text_parts) text_res = doc_serializer.post_process(text=text_res) return create_ser_result(text=text_res, span_source=item) class ImgTableAnnotationSerializerProvider(ChunkingSerializerProvider): """Custom serializer provider for chunking with image and table support.""" def get_serializer(self, doc: DoclingDocument): """Get serializer with custom picture and table serializers.""" return ChunkingDocSerializer( doc=doc, picture_serializer=AnnotationPictureSerializer(), table_serializer=MarkdownTableSerializer(), ) class SemanticChunker: """Docling-based semantic chunking for Word documents using HybridChunker.""" def __init__( self, max_tokens: int = 8000, document_id: str = None, file_name: str = None ): """ Initialize semantic chunker. Args: max_tokens: Maximum tokens per chunk (default: 8000 for text-embedding-3-large) document_id: UUID of the document file_name: Original filename """ self.logger = structlog.get_logger() self.document_id = document_id self.file_name = file_name self.max_tokens = max_tokens # Initialize OpenAI tokenizer self.tokenizer = OpenAITokenizer( tokenizer=tiktoken.encoding_for_model("text-embedding-3-large"), max_tokens=max_tokens, ) # Initialize HybridChunker with custom serializers self.chunker = HybridChunker( tokenizer=self.tokenizer, serializer_provider=ImgTableAnnotationSerializerProvider(), ) def chunk_document(self, doc: DoclingDocument, page_name: str = "page_0") -> List[Dict]: """ Chunk Word document using HybridChunker. Steps: 1. Take Docling document with full structure 2. Apply HybridChunker with custom serializers 3. Contextualize chunks 4. Build chunk records with metadata Args: doc: DoclingDocument from Docling conversion page_name: Logical page identifier Returns: List of chunk records with metadata """ # Step 1: Validate document if not doc: self.logger.warning("No document provided", file=self.file_name) return [] # Step 2: Chunk document using HybridChunker self.logger.info("Starting HybridChunker chunking", file=self.file_name) chunk_iter = self.chunker.chunk(dl_doc=doc) chunks = list(chunk_iter) self.logger.info( "HybridChunker complete", file=self.file_name, total_chunks=len(chunks) ) # Step 3: Contextualize and build chunk records chunk_records = [] for idx, chunk in enumerate(chunks): # Contextualize chunk to get final text text = self.chunker.contextualize(chunk=chunk) # Skip empty chunks if not text.strip(): continue record = self._build_chunk_record( chunk_text=text, chunk_index=idx, total_chunks=len(chunks), page_name=page_name, chunk_meta=chunk ) chunk_records.append(record) return chunk_records def _build_chunk_record( self, chunk_text: str, chunk_index: int, total_chunks: int, page_name: str, chunk_meta: Any ) -> Dict: """Build Milvus record for a chunk.""" # Extract page number from chunk metadata if available page_number = None try: from docling_core.transforms.chunker.hierarchical_chunker import DocChunk doc_chunk = DocChunk.model_validate(chunk_meta) page_numbers = [] for item in doc_chunk.meta.doc_items: if hasattr(item, 'prov') and item.prov: for prov_item in item.prov: if hasattr(prov_item, 'page_no') and prov_item.page_no is not None: page_numbers.append(prov_item.page_no) if page_numbers: page_number = min(page_numbers) except Exception as e: self.logger.debug(f"Could not extract page number: {e}") return { "document_id": self.document_id, "file_name": self.file_name, "page": page_name, "content_type": "chunk", "content": chunk_text, "metadata": { "chunk_index": chunk_index, "total_chunks": total_chunks, "char_count": len(chunk_text), "word_count": len(chunk_text.split()), "page_number": page_number, "chunking_method": "docling_hybrid", "tokenizer": "text-embedding-3-large", "max_tokens": self.max_tokens } } ``` **5. Unified Pipeline Integration** ```python # In pipeline.py async def extract_from_file(self, file_path: str) -> Optional[List[Dict]]: """ Extract from Excel or Word document. Supports: .xlsx, .xls, .xlsm, .docx, .doc, .docm """ file_ext = Path(file_path).suffix.lower() # Route to appropriate pipeline if file_ext in [".xlsx", ".xls", ".xlsm"]: return await self._extract_excel(file_path) elif file_ext in [".docx", ".doc", ".docm"]: return await self._extract_word(file_path) else: raise ValueError(f"Unsupported file format: {file_ext}") async def _extract_word(self, file_path: str, document_id: UUID) -> Optional[List[Dict]]: """ Extract from Word document using Markdown chunking. Simple pipeline: 1. Docling converts .docx → Markdown (tables, text, images all in markdown) 2. LangChain chunks the entire markdown 3. Tables are preserved in markdown table format within chunks Args: file_path: Path to .docx file document_id: UUID from documents table Returns: List of chunk records """ file_name = Path(file_path).name # Step 1: Convert to Markdown markdown = self.converter.convert_word_to_markdown(file_path) if not markdown: return None # Step 2: Semantic chunking of entire markdown chunker = SemanticChunker( chunk_size=1000, chunk_overlap=200, document_id=str(document_id), file_name=file_name ) chunk_records = chunker.chunk_document(markdown, page_name="page_0") self.logger.info( "Word extraction complete", file=file_name, total_chunks=len(chunk_records), avg_chunk_size=sum(len(r["content"]) for r in chunk_records) // len(chunk_records) if chunk_records else 0 ) return chunk_records ``` ### Integration Points **Internal Modules:** - `DocumentConverter` - Format normalization and conversion (Excel → HTML, Word → Markdown) - `WordPreprocessor` - Image/shape/comment extraction and embedding - `SemanticChunker` - Docling HybridChunker with custom serializers for Word documents - `HtmlTableExtractor` - Table extraction (Excel only) - `LlmHeaderDetector` - Header detection (Excel only) - `RecordBuilder` - JSON record generation (Excel only) **External Dependencies:** - **LibreOffice CLI** - .doc/.docm → .docx conversion - **Docling** - Word → Markdown conversion (tables included in markdown) - **LangChain** - Semantic text chunking (RecursiveCharacterTextSplitter) - **Azure OpenAI** - Image captioning (GPT-4o vision) - **python-docx** - Word document manipulation and metadata **Database Interactions:** - Documents table - Store Word files with metadata - ExtractionResults table - Store extraction status - Milvus - Vector indexing of extracted content **Event Flow:** - Celery task `extract_v2_task` triggers Word extraction - Status updates broadcast via Redis - Completion notification to API layer --- ## Implementation Stack **Runtime:** - Python 3.11+ - FastAPI 0.122 - Uvicorn (async ASGI server) - Celery 5.3+ (background tasks) - Redis 7.0+ (task queue, cache) **Document Processing:** - Docling 2.63.0+ (Word → HTML, Excel → HTML) - python-docx (Word manipulation) - openpyxl 3.1.5 (Excel manipulation) - BeautifulSoup4 4.12+ (HTML parsing) - lxml 5.0+ (parser backend) - LibreOffice 7.0+ CLI (format conversion) - docling-core (HybridChunker, custom serializers) - tiktoken (OpenAI tokenizer for chunking) **AI/ML:** - Azure OpenAI GPT-4o (header detection, image captioning) - openai 1.0+ (Python client) **Data Layer:** - PostgreSQL 18 (SQLAlchemy 2.0+, asyncpg) - Milvus 2.4+ (pymilvus) **Testing:** - pytest 8.0+ - pytest-asyncio 0.23+ **Code Quality:** - black 24.0+ (formatter, 100 char lines) - ruff 0.5+ (linter) - mypy 1.10+ (type checking) - structlog 24.0+ (logging) --- ## Technical Details ### Format Normalization Algorithm **Legacy .doc Format:** 1. Detect .doc file via extension 2. Validate file is valid Word 97-2003 format 3. Use LibreOffice CLI: `libreoffice --headless --convert-to docx` 4. Verify .docx output exists and is valid 5. Proceed with .docx pipeline **Macro-Enabled .docm Format:** 1. Detect .docm file via extension 2. Use LibreOffice CLI to convert to .docx (macros stripped) 3. Log warning about macro removal 4. Verify .docx output 5. Proceed with .docx pipeline **Timeout and Error Handling:** - 60-second timeout for LibreOffice conversion - Retry once on timeout - Fall back to V1 pipeline if conversion fails twice - Log detailed error with file characteristics ### Word Document Preprocessing Pipeline **Image Extraction and Captioning:** 1. Iterate through document relationships 2. Extract all embedded images (PNG, JPEG, EMF, WMF) 3. For each image: - Extract surrounding context (previous/next 2 paragraphs) - Send to Azure OpenAI GPT-4o vision API - Generate descriptive caption - Insert caption as text near image location 4. Preserve image references for HTML linking **Shape and Annotation Handling:** 1. Extract drawing objects (shapes, text boxes, diagrams) 2. For text-containing shapes: - Extract text content - Insert into document flow 3. For graphical shapes: - Generate description via GPT-4o vision - Insert as placeholder text **Comment Extraction:** 1. Access document comments via python-docx 2. For each comment: - Extract author, date, text - Identify commented text range - Insert marker: `[COMMENT_{id}:{author}] {text} [/COMMENT_{id}]` 3. Preserve comment threading **Note:** No table border detection needed - Docling handles table detection and converts them to markdown table format automatically. ### Semantic Chunking Strategy **Why Semantic Chunking for Word Documents?** Unlike Excel spreadsheets (which have natural row-based boundaries), Word documents contain continuous narrative text. Semantic chunking provides: - **Context Preservation:** Chunks break at natural boundaries (paragraphs, sections, sentences) - **Better Retrieval:** Semantic units improve vector search relevance - **Scalability:** Handles documents of any length without manual configuration - **Overlap Strategy:** 200-character overlap ensures context continuity across chunk boundaries **Docling HybridChunker Configuration:** ```python # HybridChunker configuration with OpenAI tokenizer import tiktoken from docling_core.transforms.chunker.hybrid_chunker import HybridChunker from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer # Initialize tokenizer for text-embedding-3-large model tokenizer = OpenAITokenizer( tokenizer=tiktoken.encoding_for_model("text-embedding-3-large"), max_tokens=8000, # Maximum tokens per chunk ) # Initialize HybridChunker with custom serializers chunker = HybridChunker( tokenizer=tokenizer, serializer_provider=ImgTableAnnotationSerializerProvider(), ) # Chunk document chunk_iter = chunker.chunk(dl_doc=doc) chunks = list(chunk_iter) # Contextualize each chunk for chunk in chunks: text = chunker.contextualize(chunk=chunk) ``` **Chunking Process:** 1. **Get Docling Document:** Docling converts entire document (tables, text, headings, lists, images preserved) 2. **Token-Based Chunking:** HybridChunker uses tiktoken tokenizer for text-embedding-3-large model 3. **Custom Serializers:** AnnotationPictureSerializer and MarkdownTableSerializer format content 4. **Contextualization:** Each chunk is contextualized to include surrounding context 5. **Tables Preserved:** Markdown tables serialized and included within chunks 6. **Images Embedded:** Picture descriptions and captions embedded in chunk text 7. **Metadata Enrichment:** Each chunk tagged with index, total count, character/word count, page number 8. **Vector Indexing:** Each chunk gets its own embedding for semantic search **Example Document Chunking:** ``` Document: 5,000 characters, 750 words ↓ Chunk 0: chars 0-1000 (paragraph boundaries) Chunk 1: chars 800-1800 (200 char overlap) Chunk 2: chars 1600-2600 (200 char overlap) Chunk 3: chars 2400-3400 (200 char overlap) Chunk 4: chars 3200-4200 (200 char overlap) Chunk 5: chars 4000-5000 (final chunk) ↓ Total: 6 chunks indexed in Milvus ``` **Integration with Preprocessing:** Images, shapes, and comments are embedded **inline** in the text before chunking: - **Images:** `[IMAGE: AI-generated caption...]` - **Shapes:** `[SHAPE: Extracted text or description...]` - **Comments:** `[COMMENT_{id}:{author}] text [/COMMENT_{id}]` This ensures they're preserved in semantic context and searchable via vector similarity. **Extraction Strategy Comparison:** | Approach | Use Case | Pros | Cons | |----------|----------|------|------| | **Component Extraction** (Excel) | Tabular data with clear row boundaries | Precise field mapping, structured queries | Requires rigid structure | | **Markdown Chunking** (Word) | Mixed content (tables + text) | Context preservation, unified approach, tables in context | Tables not queryable as structured data | **Key Decision:** Word documents use **single-mode** markdown chunking: - **Everything:** Docling → Document → HybridChunker → Contextualized chunks - **Tables:** Preserved in markdown table format within chunks - **Simpler:** No dual-mode complexity, everything is chunked uniformly ### Content Record Structure **Word Document Chunk Records (all content in markdown):** ```json { "document_id": "550e8400-e29b-41d4-a716-446655440000", "file_name": "document.docx", "page": "page_0", "content_type": "chunk", "content": "This is the first semantic chunk containing multiple paragraphs and sections. The Docling HybridChunker with OpenAI tokenizer ensures semantic boundaries are preserved, using tiktoken for accurate token counting. This maintains context and improves retrieval quality.", "metadata": { "chunk_index": 0, "total_chunks": 5, "char_count": 287, "word_count": 42, "page_number": 1, "chunking_method": "docling_hybrid", "tokenizer": "text-embedding-3-large", "max_tokens": 8000 } } ``` **Chunk with Markdown Table:** ```json { "document_id": "550e8400-e29b-41d4-a716-446655440000", "file_name": "document.docx", "page": "page_0", "content_type": "chunk", "content": "## Q1 Performance Summary\n\nThe quarterly results exceeded expectations across all metrics:\n\n| Metric | Q1 Target | Q1 Actual | % Change |\n|--------|-----------|-----------|----------|\n| Revenue | $2.5M | $3.1M | +24% |\n| Customers | 500 | 612 | +22% |\n| Satisfaction | 85% | 91% | +7% |\n\n[IMAGE: Chart showing quarterly growth trends] The chart above illustrates the upward trajectory. [COMMENT_1:Sarah Lee] Great results! Need to discuss scaling strategy [/COMMENT_1]", "metadata": { "chunk_index": 1, "total_chunks": 5, "char_count": 398, "word_count": 52, "page_number": 1, "chunking_method": "docling_hybrid", "tokenizer": "text-embedding-3-large", "max_tokens": 8000, "contains_table": true } } ``` ### Performance Considerations **LibreOffice Conversion:** - Average conversion time: 2-5 seconds per file - Timeout: 60 seconds - Parallelization: Celery workers can process multiple files concurrently **Image Captioning:** - GPT-4o vision API latency: 1-3 seconds per image - Batch images when possible (up to 10 per request) - Cache captions to avoid reprocessing **Semantic Chunking:** - Docling HybridChunker: <1 second for typical documents (<100 pages) - Average chunk size: 800-1000 characters (configured max: 1000) - Overlap: 200 characters (ensures context continuity) - No API calls required (local processing) **HTML Processing:** - BeautifulSoup parsing: <1 second for typical documents - LLM header detection: 2-4 seconds per table - Total pipeline time: 5-15 seconds per document (depending on images and tables) ### Security Considerations **Macro Safety:** - .docm macros are NOT executed - LibreOffice conversion strips macros - Log macro presence for audit trail **File Validation:** - Validate file size (<100MB) - Check MIME type matches extension - Scan for malformed ZIP structure (.docx is ZIP-based) **Injection Prevention:** - Sanitize extracted text before LLM calls - Escape HTML entities in content extraction - Validate image data before captioning --- ## Development Setup **Prerequisites:** ```bash # System dependencies sudo apt-get install libreoffice # Ubuntu/Debian brew install libreoffice # macOS # Verify installation libreoffice --version ``` **Environment Setup:** ```bash # Clone and navigate to project cd /home/hieutt50/projects/DSOL # Install Python dependencies (includes python-docx, docling) uv sync --all-extras # Configure environment cp .env.example .env # Edit .env with Azure OpenAI credentials # Start infrastructure make services # PostgreSQL, Redis, Milvus # Run migrations make migrate # Start development servers make dev # API + Celery worker ``` **Running Tests:** ```bash # Unit tests for Word processing uv run pytest tests/test_extraction_v2/test_word_conversion.py -v # Integration tests for all Word files uv run pytest tests/test_extraction_v2/test_word_comprehensive.py -v # Full test suite uv run pytest tests/test_extraction_v2/ -v --cov=src/extraction_v2 ``` --- ## Acceptance Criteria **Overall Success:** 1. ✅ All 51 Word documents in CustomerDocument/ discovered and tracked 2. ✅ .doc and .docm files successfully converted to .docx 3. ✅ Images extracted and captioned via AI (embedded inline in markdown) 4. ✅ Shapes, annotations, and comments extracted (embedded inline in markdown) 5. ✅ Docling converts Word → Markdown (tables preserved in markdown format) 6. ✅ Full document chunked using Docling HybridChunker with OpenAI tokenizer 7. ✅ Chunk configuration: 1000 chars max, 200 char overlap 8. ✅ Tables preserved in chunks as markdown tables (no separate extraction) 9. ✅ ≥85% extraction success rate (V2 + V1 fallback) 10. ✅ Batch processor includes Word files in status tracking 11. ✅ Comprehensive test suite passing (markdown chunking validation included) --- **End of Technical Specification**