# Epic: PDF and PowerPoint Document Processing Support

**Epic ID:** pdf-pptx-support
**Project:** DSOL
**Created:** 2025-12-05
**Status:** Ready for Development

---

## Epic Goal

Extend the V2 extraction pipeline to process PDF and PowerPoint documents with full feature parity to Word document processing, achieving ≥85% extraction success rate through unified Markdown conversion and semantic chunking.

---

## Epic Scope

**In Scope:**
- PDF support: All types (text-based, scanned with OCR, mixed content)
- PowerPoint format normalization for legacy .ppt and macro-enabled .pptm files
- PowerPoint speaker notes extraction
- Docling-based PDF → Markdown and PowerPoint → Markdown conversion
- Docling HybridChunker with OpenAI tokenizer (tiktoken) and custom serializers
- Custom serializers for images and tables (ImgTableAnnotationSerializerProvider)
- Batch processing updates for PDF/PowerPoint file discovery
- Comprehensive testing suite
- Documentation and troubleshooting guides

**Out of Scope:**
- Advanced PDF form field extraction (basic text extraction only)
- PowerPoint animation or transition extraction
- Video/audio embedded in PowerPoint
- PDF password protection handling (requires manual unlock)
- PDF digital signature validation
- Macro execution (.pptm macros not run)

---

## Success Criteria

1. ✅ All PDF files discovered and tracked (text-based, scanned, mixed)
2. ✅ Format normalization working (.ppt/.pptm → .pptx via LibreOffice)
3. ✅ Speaker notes extracted and embedded (PowerPoint)
4. ✅ Docling converts PDF → Markdown, PowerPoint → Markdown
5. ✅ Documents chunked using Docling HybridChunker with OpenAI tokenizer
6. ✅ Custom serializers handle images and tables within chunks
7. ✅ ≥85% extraction success rate (V2 + V1 fallback if applicable)
8. ✅ Batch processor tracks PDF/PowerPoint files with metrics
9. ✅ Automated test suite validates extraction quality
10. ✅ Documentation updated with PDF/PowerPoint support

---

## Story Breakdown

### Story 1: PDF Document Processing
**File:** `story-pdf-pptx-support-1.md`
**Focus:** PDF → Markdown conversion with Docling

**Key Tasks:**
- Add PDF format detection to document_converter.py
- Implement _convert_pdf_to_markdown() method using Docling
- Add PDF file validation (<100MB, header check)
- Handle all PDF types: text-based, scanned (OCR), mixed
- Integrate with SemanticChunker and custom serializers
- Configure ImgTableAnnotationSerializerProvider for image/table handling

**Acceptance:** All PDF files successfully converted to Markdown with Docling, images and tables handled by custom serializers during chunking

---

### Story 2: PowerPoint Format Normalization & Speaker Notes
**File:** `story-pdf-pptx-support-2.md`
**Focus:** .ppt/.pptm → .pptx conversion and speaker notes extraction

**Key Tasks:**
- Implement .ppt/.pptm → .pptx conversion via LibreOffice CLI
- Create PptxPreprocessor class for speaker notes extraction
- Extract speaker notes and embed inline in slides
- Implement file locking for concurrent LibreOffice access
- 60-second timeout with retry logic
- Validate converted files

**Acceptance:** All PowerPoint files (.pptx, .ppt, .pptm) successfully normalized, speaker notes embedded inline

---

### Story 3: PowerPoint Markdown Conversion & Pipeline Integration
**File:** `story-pdf-pptx-support-3.md`
**Focus:** PowerPoint → Markdown conversion and pipeline integration

**Key Tasks:**
- Implement _convert_pptx_to_markdown() method using Docling
- Integrate with Docling for PPTX → Markdown conversion
- Add process_pdf_document() to pipeline.py
- Add process_pptx_document() to pipeline.py
- Create sync versions of async methods
- Integrate with SemanticChunker and custom serializers
- Configure ImgTableAnnotationSerializerProvider for both PDF and PowerPoint
- Route PDF/PowerPoint through semantic chunker
- Update document_converter.py routing logic
- Handle slide structure preservation

**Acceptance:** PDF and PowerPoint documents fully integrated into V2 pipeline with Markdown output

---

### Story 4: Testing, Batch Processing & Documentation
**File:** `story-pdf-pptx-support-4.md`
**Focus:** Comprehensive testing and integration

**Key Tasks:**
- Create test_pdf_conversion.py (unit tests)
- Create test_pptx_conversion.py (unit tests)
- Create test_pdf_preprocessing.py (image captioning tests)
- Create test_pptx_preprocessing.py (preprocessing tests)
- Create test_pdf_pptx_comprehensive.py (integration tests)
- Update batch processing glob patterns (.pdf/.pptx/.ppt/.pptm)
- Success rate validation (≥85%)
- Error categorization for format-specific issues
- Update README with PDF/PowerPoint support
- Update extraction-pipeline-v2-tech-spec.md
- Create troubleshooting guide

**Acceptance:** Test suite passes, batch processor tracks PDF/PowerPoint, ≥85% success rate, docs updated

---

## Implementation Sequence

```
Story 1 (PDF Processing)      Story 2 (PowerPoint Preprocessing)
       ↓                                    ↓
       └───────────────┬────────────────────┘
                       ↓
           Story 3 (Pipeline Integration)
                       ↓
           Story 4 (Testing & Documentation)
                       ↓
       Validation: Run full test suite, verify 85% success rate
```

**Dependencies:**
- Story 1 and Story 2 can run in parallel (independent)
- Story 3 depends on Story 1 and Story 2 (needs both converters)
- Story 4 depends on Story 3 (needs complete pipeline)

---

## Technical Context

**From Tech-Spec:**
- PDF: All types supported (text-based, scanned, mixed)
- PowerPoint: .pptx, .ppt, .pptm formats
- Target: ≥85% extraction success rate
- Follow Word document pattern (Markdown conversion + semantic chunking)

**Stack:**
- Python 3.11+, FastAPI 0.122, Celery 5.3+
- Docling 2.63.0+ (PDF → Markdown, PowerPoint → Markdown)
- docling-core (HybridChunker, ImgTableAnnotationSerializerProvider)
- tiktoken (OpenAI tokenizer for text-embedding-3-large)
- python-pptx (PowerPoint speaker notes extraction) - NEW DEPENDENCY
- LibreOffice 7.0+ CLI (format conversion)
- PostgreSQL, Milvus 2.4+

**Key Files:**
- `src/extraction_v2/document_converter.py` (MODIFY)
- `src/extraction_v2/pptx_preprocessor.py` (CREATE - speaker notes only)
- `src/extraction_v2/pipeline.py` (MODIFY)
- `src/extraction_v2/semantic_chunker.py` (REUSE - existing with ImgTableAnnotationSerializerProvider)
- `src/api/routes/documents.py` (MODIFY - file validation)
- `src/workers/extraction_v2_tasks.py` (MODIFY)
- `batch_extract_v2.py` (MODIFY - if exists)

---

## Epic Value

**Business Impact:**
- Support common enterprise document formats (PDF, PowerPoint)
- Complete document coverage for production readiness
- Enable processing of presentations and reports
- Extract valuable structured data from slides and PDFs
- Competitive parity with comprehensive document processing systems

**Developer Impact:**
- Reusable preprocessing patterns across document types
- Unified Markdown conversion strategy (Word, PDF, PowerPoint)
- Consistent semantic chunking approach
- Comprehensive test coverage
- Clear troubleshooting guides

---

## Epic Notes

- **Unified Strategy:** All text documents (Word, PDF, PowerPoint) → Markdown → HybridChunker with custom serializers
- **Excel Exception:** Excel remains HTML-based for table structure preservation
- **LibreOffice Pattern:** Reuses .doc/.docm conversion approach for .ppt/.pptm
- **Image & Table Handling:** Custom serializers (ImgTableAnnotationSerializerProvider) handle images and tables during chunking - NO preprocessing needed
- **Simpler Approach:** Docling handles all image/table extraction automatically via custom serializers
- **Docling Power:** Handles PDF OCR automatically, PowerPoint slide structure natively, images/tables via serializers
- **Speaker Notes:** PowerPoint-specific feature, embedded inline via PptxPreprocessor as `[SPEAKER NOTES]: {text}`
- **Token Limits:** 8000 tokens max per chunk (text-embedding-3-large model)
- **No Image Captioning:** Unlike Word documents, PDF/PowerPoint rely on Docling's native image handling through serializers

---

## Risk Mitigation

**Risk 1: LibreOffice Conversion Failures**
- Mitigation: 60-second timeout, retry logic, file locking, graceful fallback

**Risk 2: PDF Complexity (Scanned, Mixed)**
- Mitigation: Docling handles OCR internally, validate with diverse test files

**Risk 3: Image/Table Serialization**
- Mitigation: Use proven ImgTableAnnotationSerializerProvider from Word pipeline, extensive testing

**Risk 4: Speaker Notes Extraction**
- Mitigation: Use python-pptx library, well-documented API, extensive testing

---

**Epic Status:** Ready for Sprint Planning
**Estimated Stories:** 4 stories
**Story Size:** Medium (2-5 days each depending on complexity)

