# User Story: Format Normalization & LibreOffice Integration

**Story ID:** story-word-support-1
**Epic:** Word Document Processing Support
**Created:** 2025-12-01
**Status:** Done
**Story Points:** 3
**Priority:** High

---

## User Story

**As a** data engineer
**I want** legacy .doc and macro-enabled .docm files automatically converted to .docx format
**So that** the V2 pipeline can process all Word document formats using Docling's .docx support

---

## Context

**Current Situation:**
- V2 pipeline only handles modern .docx format
- CustomerDocument/ contains 21 .doc files (legacy Word 97-2003)
- CustomerDocument/ contains 28 .docm files (macro-enabled)
- Docling requires .docx format for Word → Markdown conversion

**Problem:**
Cannot process 49/51 Word files (96%) without format normalization.

**Solution:**
Implement LibreOffice CLI integration to convert .doc and .docm files to .docx before Docling processing, following the existing pattern from Excel (.xls/.xlsm → .xlsx).

---

## Technical Details

**File to Create:**
`src/extraction_v2/document_to_html_converter.py`

**Implementation Pattern (from Excel):**
```python
# Current Excel pattern in excel_to_html_converter.py
def _convert_to_xlsx_via_libreoffice(self, file_path: Path) -> Path:
    """Convert .xls or .xlsm to .xlsx using LibreOffice CLI."""
    # ... LibreOffice subprocess call ...
```

**New Word Pattern:**
```python
# In document_to_html_converter.py

class DocumentToHtmlConverter:
    """Converts Excel and Word documents to HTML using Docling."""

    def __init__(self):
        self.converter = DocumentConverter(
            format_options={
                InputFormat.XLSX: ExcelFormatOption(),
                InputFormat.DOCX: DocxFormatOption()  # Add Word support
            }
        )
        self.logger = structlog.get_logger()

    def convert_to_html(self, file_path: str) -> Optional[str]:
        """
        Convert document to HTML.

        Supports: .xlsx, .xls, .xlsm, .docx, .doc, .docm
        """
        file_path = Path(file_path)
        file_ext = file_path.suffix.lower()

        # Route to appropriate conversion
        if file_ext in [".xlsx", ".xls", ".xlsm"]:
            return self._convert_excel_to_html(file_path)
        elif file_ext in [".docx", ".doc", ".docm"]:
            return self._convert_word_to_markdown(file_path)
        else:
            raise ValueError(f"Unsupported format: {file_ext}")

    def _convert_word_to_markdown(self, file_path: Path) -> Optional[str]:
        """Convert Word document to Markdown."""
        # Step 1: Format normalization
        if file_path.suffix in [".doc", ".docm"]:
            file_path = self._convert_doc_to_docx(file_path)

        # Step 2: Docling conversion
        result = self.converter.convert(str(file_path))

        if result and result.document:
            markdown = result.document.export_to_markdown()
            self.logger.info("Word → Markdown conversion successful", file=file_path.name)
            return markdown
        else:
            self.logger.error("Word → Markdown conversion failed", file=file_path.name)
            return None

    def _convert_doc_to_docx(self, file_path: Path) -> Path:
        """
        Convert .doc or .docm to .docx using LibreOffice CLI.

        Handles:
        - .doc (Word 97-2003) → .docx
        - .docm (macro-enabled) → .docx (macros stripped)
        """
        # Validate LibreOffice is installed
        if not shutil.which("libreoffice"):
            raise RuntimeError(
                "LibreOffice not installed. Required for .doc/.docm conversion. "
                "Install: apt-get install libreoffice (Ubuntu) or brew install libreoffice (macOS)"
            )

        # Create temp directory for conversion
        temp_dir = Path(tempfile.mkdtemp(prefix="docx_convert_"))
        output_path = temp_dir / f"{file_path.stem}.docx"

        try:
            self.logger.info(
                "Converting to .docx",
                input=file_path.name,
                format=file_path.suffix
            )

            # Run LibreOffice conversion
            result = subprocess.run(
                [
                    "libreoffice",
                    "--headless",
                    "--convert-to", "docx",
                    "--outdir", str(temp_dir),
                    str(file_path)
                ],
                check=True,
                capture_output=True,
                timeout=60,
                text=True
            )

            # Verify conversion succeeded
            converted_file = temp_dir / f"{file_path.stem}.docx"
            if not converted_file.exists():
                raise RuntimeError(
                    f"Conversion failed: {file_path.name}. "
                    f"LibreOffice output: {result.stdout}"
                )

            # Validate converted file
            if converted_file.stat().st_size == 0:
                raise RuntimeError(f"Conversion produced empty file: {file_path.name}")

            self.logger.info(
                "Conversion successful",
                input=file_path.name,
                output=converted_file.name,
                size_mb=converted_file.stat().st_size / (1024 * 1024)
            )

            return converted_file

        except subprocess.TimeoutExpired:
            self.logger.error("Conversion timeout", file=file_path.name, timeout=60)
            raise RuntimeError(f"Conversion timeout (60s): {file_path.name}")

        except subprocess.CalledProcessError as e:
            self.logger.error(
                "LibreOffice conversion error",
                file=file_path.name,
                stderr=e.stderr
            )
            raise RuntimeError(f"LibreOffice error: {e.stderr}")

        except Exception as e:
            self.logger.error("Unexpected conversion error", file=file_path.name, error=str(e))
            raise
```

**File Validation:**
```python
def _validate_word_file(self, file_path: Path) -> tuple[bool, Optional[str]]:
    """Validate Word file before conversion."""
    # Check exists
    if not file_path.exists():
        return False, "File not found"

    # Check size (<100MB)
    size_mb = file_path.stat().st_size / (1024 * 1024)
    if size_mb > 100:
        return False, f"File too large: {size_mb:.1f}MB"

    # Check extension
    ext = file_path.suffix.lower()
    if ext not in ['.doc', '.docx', '.docm']:
        return False, f"Unsupported format: {ext}"

    # For .docx/.docm: verify ZIP structure
    if ext in ['.docx', '.docm']:
        try:
            import zipfile
            with zipfile.ZipFile(file_path, 'r') as zf:
                if 'word/document.xml' not in zf.namelist():
                    return False, "Invalid Word document structure"
        except zipfile.BadZipFile:
            return False, "Corrupted Word file (not a valid ZIP)"

    return True, None
```

---

## Implementation Steps

1. **Rename and extend excel_to_html_converter.py**
   - Rename to `document_to_html_converter.py`
   - Update class name: `ExcelToHtmlConverter` → `DocumentToHtmlConverter`
   - Add Word format detection logic
   - Keep existing Excel conversion methods

2. **Implement _convert_doc_to_docx() method**
   - Check LibreOffice installation
   - Create temp directory for conversion
   - Run subprocess with timeout (60s)
   - Validate output file exists and is non-empty
   - Return path to converted .docx file
   - Clean up temp files on error

3. **Implement _convert_word_to_markdown() method**
   - Detect Word file extension
   - Call _convert_doc_to_docx() if needed
   - Use Docling to convert .docx → Markdown
   - Return Markdown string

4. **Add file validation**
   - Check file existence
   - Validate file size (<100MB)
   - Verify ZIP structure for .docx/.docm
   - Return clear error messages

5. **Update imports and configuration**
   - Add DocxFormatOption to Docling converter
   - Update structlog logger calls
   - Add error handling for missing LibreOffice

6. **Test conversion pipeline**
   ```bash
   # Test .doc conversion
   python -c "
   from src.extraction_v2.document_to_html_converter import DocumentToHtmlConverter
   converter = DocumentToHtmlConverter()
   markdown = converter.convert_to_markdown('CustomerDocument/.../file.doc')
   print(f'Markdown length: {len(markdown)}')
   assert markdown is not None
   "

   # Test .docm conversion
   python -c "
   from src.extraction_v2.document_to_html_converter import DocumentToHtmlConverter
   converter = DocumentToHtmlConverter()
   markdown = converter.convert_to_markdown('CustomerDocument/.../file.docm')
   print(f'Markdown length: {len(markdown)}')
   assert markdown is not None
   "
   ```

---

## Acceptance Criteria

1. ✅ DocumentToHtmlConverter class created
   - Supports both Excel and Word formats
   - Routes to appropriate conversion method

2. ✅ .doc → .docx conversion working
   - LibreOffice CLI integration functional
   - All 21 .doc files convert successfully
   - Converted files validated

3. ✅ .docm → .docx conversion working
   - Macro stripping logged as warning
   - All 28 .docm files convert successfully
   - Macros not executed (security)

4. ✅ Error handling robust
   - Timeout detection (60s)
   - Missing LibreOffice detected with clear message
   - Corrupted files identified
   - Clear error messages for all failure modes

5. ✅ File validation implemented
   - Size limits enforced (<100MB)
   - ZIP structure verified for .docx/.docm
   - Unsupported formats rejected

6. ✅ Logging comprehensive
   - Conversion start/success/failure logged
   - File sizes and formats logged
   - Performance metrics tracked

---

## Testing Strategy

**Unit Tests:**
```python
# tests/test_extraction_v2/test_word_conversion.py

def test_convert_doc_to_docx_success():
    converter = DocumentToHtmlConverter()
    doc_path = Path("CustomerDocument/test.doc")

    docx_path = converter._convert_doc_to_docx(doc_path)

    assert docx_path.exists()
    assert docx_path.suffix == ".docx"
    assert docx_path.stat().st_size > 0

def test_convert_docm_to_docx_strips_macros():
    converter = DocumentToHtmlConverter()
    docm_path = Path("CustomerDocument/test.docm")

    docx_path = converter._convert_doc_to_docx(docm_path)

    assert docx_path.exists()
    assert docx_path.suffix == ".docx"
    # Verify macros stripped (check for vbaProject.bin absence)
    import zipfile
    with zipfile.ZipFile(docx_path) as zf:
        assert 'word/vbaProject.bin' not in zf.namelist()

def test_conversion_timeout():
    converter = DocumentToHtmlConverter()
    # Create a large/complex file that times out
    with pytest.raises(RuntimeError, match="timeout"):
        converter._convert_doc_to_docx(large_file_path)

def test_missing_libreoffice():
    with patch('shutil.which', return_value=None):
        converter = DocumentToHtmlConverter()
        with pytest.raises(RuntimeError, match="LibreOffice not installed"):
            converter._convert_doc_to_docx(Path("test.doc"))

def test_validate_word_file():
    converter = DocumentToHtmlConverter()

    # Valid file
    valid, msg = converter._validate_word_file(Path("test.docx"))
    assert valid is True

    # File too large
    with patch('pathlib.Path.stat') as mock_stat:
        mock_stat.return_value.st_size = 200 * 1024 * 1024  # 200MB
        valid, msg = converter._validate_word_file(Path("large.docx"))
        assert valid is False
        assert "too large" in msg
```

**Integration Tests:**
```bash
# Test with real files from CustomerDocument/
uv run pytest tests/test_extraction_v2/test_word_conversion.py::test_convert_all_doc_files -v
uv run pytest tests/test_extraction_v2/test_word_conversion.py::test_convert_all_docm_files -v
```

---

## Dependencies

**External Dependencies:**
- LibreOffice 7.0+ CLI (system dependency)
- Docling 2.63.0+ with DocxFormatOption
- python-docx (for validation)

**Internal Dependencies:**
- Existing `ExcelToHtmlConverter` code (reuse pattern)
- Docling DocumentConverter infrastructure
- structlog logging

**System Requirements:**
```bash
# Ubuntu/Debian
sudo apt-get install libreoffice

# macOS
brew install libreoffice

# Verify
libreoffice --version
```

**Blockers:**
- None - can start immediately

---

## Definition of Done

- [ ] document_to_html_converter.py created (renamed from excel_to_html_converter.py)
- [ ] _convert_doc_to_docx() method implemented
- [ ] _convert_word_to_html() method implemented
- [ ] File validation implemented
- [ ] LibreOffice detection and error handling
- [ ] All unit tests passing
- [ ] Integration tests with real .doc/.docm files passing
- [ ] All 21 .doc files convert successfully
- [ ] All 28 .docm files convert successfully
- [ ] Logging comprehensive and clear
- [ ] Code reviewed
- [ ] Ready for Story 2 (preprocessing)

---

## Reference

**Tech-Spec:** `docs/feats/docx_support/tech-spec.md`
**Epic:** `docs/feats/docx_support/epics.md`
**Pattern Reference:** `src/extraction_v2/excel_to_html_converter.py` (existing Excel conversion)
**Key Sections:**
- Technical Approach: Format Normalization (tech-spec.md)
- Implementation Stack (tech-spec.md)

---

**Story Status:** Ready for Development
