# User Story: PDF Document Processing

**Story ID:** story-pdf-pptx-support-1
**Epic:** PDF and PowerPoint Document Processing Support
**Created:** 2025-12-05
**Status:** Done
**Story Points:** 2
**Priority:** High

---

## User Story

**As a** data engineer
**I want** PDF documents (text-based, scanned, and mixed) automatically converted to Markdown and chunked with custom serializers
**So that** the V2 pipeline can process all PDF types with images and tables handled seamlessly during chunking

---

## Context

**Current Situation:**
- V2 pipeline handles Excel (.xlsx, .xls, .xlsm) and Word (.docx, .doc, .docm) documents
- Word documents use Docling HybridChunker with ImgTableAnnotationSerializerProvider
- No PDF support currently exists
- PDFs are common in enterprise environments (reports, forms, documents)
- Docling 2.63.0+ has native PDF support with OCR capability

**Problem:**
Cannot process PDF documents, which are prevalent in enterprise document collections.

**Solution:**
Implement PDF → Markdown conversion using Docling, integrate with SemanticChunker using ImgTableAnnotationSerializerProvider for automatic image/table handling during chunking.

---

## Technical Details

**Files to Modify:**
- `src/extraction_v2/document_converter.py` - Add PDF conversion method
- `src/api/routes/documents.py` - Add .pdf to accepted formats

**PDF Conversion Implementation:**
```python
# In document_converter.py

def _convert_pdf_to_markdown(self, file_path: str) -> Optional[str]:
    """
    Convert PDF to Markdown using Docling.

    Pipeline:
    1. Validate file
    2. Convert PDF to Markdown (via Docling)
    
    Images and tables are handled by custom serializers during chunking phase.
    """
    file_path_obj = Path(file_path)

    try:
        # Validation
        is_valid, error_msg = self._validate_pdf_file(file_path_obj)
        if not is_valid:
            logger.error(f"PDF validation failed: {error_msg}")
            return None

        # Docling conversion (no preprocessing needed)
        logger.info(f"Converting PDF to Markdown via Docling: {file_path_obj}")
        _ensure_event_loop()
        result = self.converter.convert(str(file_path_obj))

        if not result or not result.document:
            raise RuntimeError("Docling conversion returned empty result")

        markdown = result.document.export_to_markdown()
        logger.info(f"Successfully converted PDF to Markdown ({len(markdown)} chars)")
        return markdown

    except Exception as e:
        logger.error(f"Docling PDF conversion failed: {e}")
        return None

def _validate_pdf_file(self, file_path: Path) -> tuple[bool, Optional[str]]:
    """Validate PDF file before conversion."""
    if not file_path.exists():
        return False, "File not found"

    size_mb = file_path.stat().st_size / (1024 * 1024)
    if size_mb > 100:
        return False, f"File too large: {size_mb:.1f}MB (max 100MB)"

    ext = file_path.suffix.lower()
    if ext != '.pdf':
        return False, f"Not a PDF file: {ext}"

    # Verify PDF structure
    try:
        with open(file_path, 'rb') as f:
            header = f.read(5)
            if not header.startswith(b'%PDF-'):
                return False, "Invalid PDF file (missing PDF header)"
    except Exception as e:
        return False, f"File validation error: {str(e)}"

    return True, None
```

**Update Routing in convert_to_html():**
```python
def convert_to_html(self, file_path: str) -> Optional[str]:
    """
    Convert document to HTML/Markdown.

    Routes:
    - Excel → HTML
    - Word → Markdown
    - PDF → Markdown (NEW)
    """
    file_ext = Path(file_path).suffix.lower()

    if file_ext in ['.xlsx', '.xls', '.xlsm']:
        return self._convert_excel_to_html(file_path)
    elif file_ext in ['.docx', '.doc', '.docm']:
        return self._convert_word_to_markdown(file_path)
    elif file_ext == '.pdf':
        return self._convert_pdf_to_markdown(file_path)
    else:
        logger.error(f"Unsupported file format: {file_ext}")
        return None
```

**Integration with SemanticChunker (Already Exists):**
```python
# The existing SemanticChunker already uses ImgTableAnnotationSerializerProvider
# No changes needed - it will automatically handle PDF images and tables

# In semantic_chunker.py (existing code):
from docling_core.transforms.chunker import HybridChunker
from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer
from src.extraction_v2.custom_serializers import ImgTableAnnotationSerializerProvider

chunker = HybridChunker(
    tokenizer=OpenAITokenizer(...),
    serializer_provider=ImgTableAnnotationSerializerProvider()  # Handles images/tables
)
```

---

## Implementation Steps

1. **Add _convert_pdf_to_markdown() method**
   - Implement in document_converter.py
   - Use Docling DocumentConverter
   - Handle PDF → Markdown conversion
   - NO preprocessing needed

2. **Add _validate_pdf_file() method**
   - Check file existence
   - Validate size (<100MB)
   - Verify PDF header structure
   - Return clear error messages

3. **Update convert_to_html() routing**
   - Add PDF to format detection
   - Route .pdf files to _convert_pdf_to_markdown()

4. **Update API file validation**
   - Modify src/api/routes/documents.py
   - Add .pdf to accepted MIME types
   - Update file validation logic

5. **Test PDF conversion**
   ```bash
   # Test text-based PDF
   python -c "
   from src.extraction_v2.document_converter import DocumentConverter
   converter = DocumentConverter()
   markdown = converter._convert_pdf_to_markdown('test_text.pdf')
   assert markdown is not None
   print(f'Markdown length: {len(markdown)}')
   "

   # Test scanned PDF (with OCR)
   python -c "
   from src.extraction_v2.document_converter import DocumentConverter
   converter = DocumentConverter()
   markdown = converter._convert_pdf_to_markdown('test_scanned.pdf')
   assert markdown is not None
   print(f'Markdown length: {len(markdown)}')
   "
   ```

---

## Acceptance Criteria

1. ✅ _convert_pdf_to_markdown() method implemented
   - Uses Docling for conversion
   - Returns Markdown string
   - No preprocessing required

2. ✅ PDF → Markdown conversion working
   - Text-based PDFs convert successfully
   - Scanned PDFs processed with OCR (Docling handles)
   - Mixed-content PDFs handled

3. ✅ File validation robust
   - Size limits enforced (<100MB)
   - PDF header verification
   - Clear error messages

4. ✅ API accepts PDF uploads
   - .pdf added to accepted formats
   - MIME type validation updated

5. ✅ Images and tables handled by serializers
   - No manual image extraction needed
   - ImgTableAnnotationSerializerProvider handles automatically
   - Images preserved within chunks

6. ✅ Logging comprehensive
   - Conversion start/success/failure logged
   - Performance metrics recorded

---

## Testing Strategy

**Unit Tests:**
```python
# tests/test_extraction_v2/test_pdf_conversion.py

def test_pdf_text_to_markdown():
    converter = DocumentConverter()
    markdown = converter._convert_pdf_to_markdown("test_text.pdf")
    assert markdown is not None
    assert len(markdown) > 0

def test_pdf_scanned_with_ocr():
    converter = DocumentConverter()
    markdown = converter._convert_pdf_to_markdown("test_scanned.pdf")
    assert markdown is not None
    # Docling handles OCR automatically

def test_pdf_validation():
    converter = DocumentConverter()

    # Valid PDF
    valid, msg = converter._validate_pdf_file(Path("test.pdf"))
    assert valid is True

    # Too large
    with patch('pathlib.Path.stat') as mock_stat:
        mock_stat.return_value.st_size = 200 * 1024 * 1024
        valid, msg = converter._validate_pdf_file(Path("large.pdf"))
        assert valid is False
        assert "too large" in msg

def test_pdf_images_handled_by_serializer():
    """Verify images are handled during chunking, not preprocessing."""
    pipeline = ExtractionPipelineV2()
    result = pipeline.process_pdf_document_sync(uuid.uuid4(), "test_with_images.pdf")
    
    assert result.success is True
    # Images should be present in chunks via serializer
    # No separate image metadata files needed
```

---

## Dependencies

**New Python Dependencies:**
- None (Docling already installed)

**External Dependencies:**
- Docling 2.63.0+ (PDF conversion with OCR)

**Internal Dependencies:**
- DocumentConverter infrastructure
- SemanticChunker with ImgTableAnnotationSerializerProvider (existing)
- structlog logging

**Blockers:**
- None - can start immediately

---

## Definition of Done

- [x] _convert_pdf_to_markdown() method implemented
- [x] _validate_pdf_file() method implemented
- [x] convert_to_html() routing updated for PDF
- [x] API updated to accept .pdf files
- [x] All unit tests passing
- [x] Text-based PDFs convert successfully
- [x] Scanned PDFs process with OCR
- [x] Mixed-content PDFs handled
- [x] Images/tables handled by serializers (no preprocessing)
- [x] Logging comprehensive
- [x] Code reviewed
- [x] Ready for pipeline integration (Story 3)

---

## Reference

**Tech-Spec:** `docs/feats/pdf_pptx_support/tech-spec.md`
**Epic:** `docs/feats/pdf_pptx_support/epics.md`
**Pattern:** Word document conversion (uses same serializers)
**Key Sections:**
- Technical Approach: PDF Document Processing
- Implementation Stack
- Content Record Structure

---

**Story Status:** Done

## Implementation Notes

**Completed:** 2025-12-05

**Files Modified:**
- `src/extraction_v2/document_converter.py` (+78 lines)
  - Added `_convert_pdf_to_markdown()` method using Docling
  - Added `_validate_pdf_file()` validation method
  - Updated `convert_to_html()` routing to handle .pdf files
  - Added `PdfFormatOption` to DocumentConverter initialization

- `src/services/file_validator.py` (+15 lines)
  - Added `PDF_MAGIC_BYTES` constant (`%PDF-`)
  - Updated `validate_file_extension()` to accept .pdf
  - Updated `validate_magic_bytes()` to validate PDF signature

- `tests/test_extraction_v2/test_pdf_conversion.py` (165 lines, NEW)
  - 15 comprehensive unit tests covering all scenarios
  - All tests passing

**Test Results:**
- ✅ 15/15 PDF conversion tests PASSED
- ✅ Real-world PDF tested with customer document (Japanese/Chinese)
- ✅ OCR functionality verified (RapidOCR PP-OCRv4 models)
- ✅ No regressions in existing tests

**Git Commit:**
- Commit: 6456d15
- Branch: feat/pdf-support
- Message: "feat: Add PDF document processing support to V2 extraction pipeline"
