# Epic: PDF and PowerPoint Document Processing Support **Epic ID:** pdf-pptx-support **Project:** DSOL **Created:** 2025-12-05 **Status:** Ready for Development --- ## Epic Goal Extend the V2 extraction pipeline to process PDF and PowerPoint documents with full feature parity to Word document processing, achieving ≥85% extraction success rate through unified Markdown conversion and semantic chunking. --- ## Epic Scope **In Scope:** - PDF support: All types (text-based, scanned with OCR, mixed content) - PowerPoint format normalization for legacy .ppt and macro-enabled .pptm files - PowerPoint speaker notes extraction - Docling-based PDF → Markdown and PowerPoint → Markdown conversion - Docling HybridChunker with OpenAI tokenizer (tiktoken) and custom serializers - Custom serializers for images and tables (ImgTableAnnotationSerializerProvider) - Batch processing updates for PDF/PowerPoint file discovery - Comprehensive testing suite - Documentation and troubleshooting guides **Out of Scope:** - Advanced PDF form field extraction (basic text extraction only) - PowerPoint animation or transition extraction - Video/audio embedded in PowerPoint - PDF password protection handling (requires manual unlock) - PDF digital signature validation - Macro execution (.pptm macros not run) --- ## Success Criteria 1. ✅ All PDF files discovered and tracked (text-based, scanned, mixed) 2. ✅ Format normalization working (.ppt/.pptm → .pptx via LibreOffice) 3. ✅ Speaker notes extracted and embedded (PowerPoint) 4. ✅ Docling converts PDF → Markdown, PowerPoint → Markdown 5. ✅ Documents chunked using Docling HybridChunker with OpenAI tokenizer 6. ✅ Custom serializers handle images and tables within chunks 7. ✅ ≥85% extraction success rate (V2 + V1 fallback if applicable) 8. ✅ Batch processor tracks PDF/PowerPoint files with metrics 9. ✅ Automated test suite validates extraction quality 10. ✅ Documentation updated with PDF/PowerPoint support --- ## Story Breakdown ### Story 1: PDF Document Processing **File:** `story-pdf-pptx-support-1.md` **Focus:** PDF → Markdown conversion with Docling **Key Tasks:** - Add PDF format detection to document_converter.py - Implement _convert_pdf_to_markdown() method using Docling - Add PDF file validation (<100MB, header check) - Handle all PDF types: text-based, scanned (OCR), mixed - Integrate with SemanticChunker and custom serializers - Configure ImgTableAnnotationSerializerProvider for image/table handling **Acceptance:** All PDF files successfully converted to Markdown with Docling, images and tables handled by custom serializers during chunking --- ### Story 2: PowerPoint Format Normalization & Speaker Notes **File:** `story-pdf-pptx-support-2.md` **Focus:** .ppt/.pptm → .pptx conversion and speaker notes extraction **Key Tasks:** - Implement .ppt/.pptm → .pptx conversion via LibreOffice CLI - Create PptxPreprocessor class for speaker notes extraction - Extract speaker notes and embed inline in slides - Implement file locking for concurrent LibreOffice access - 60-second timeout with retry logic - Validate converted files **Acceptance:** All PowerPoint files (.pptx, .ppt, .pptm) successfully normalized, speaker notes embedded inline --- ### Story 3: PowerPoint Markdown Conversion & Pipeline Integration **File:** `story-pdf-pptx-support-3.md` **Focus:** PowerPoint → Markdown conversion and pipeline integration **Key Tasks:** - Implement _convert_pptx_to_markdown() method using Docling - Integrate with Docling for PPTX → Markdown conversion - Add process_pdf_document() to pipeline.py - Add process_pptx_document() to pipeline.py - Create sync versions of async methods - Integrate with SemanticChunker and custom serializers - Configure ImgTableAnnotationSerializerProvider for both PDF and PowerPoint - Route PDF/PowerPoint through semantic chunker - Update document_converter.py routing logic - Handle slide structure preservation **Acceptance:** PDF and PowerPoint documents fully integrated into V2 pipeline with Markdown output --- ### Story 4: Testing, Batch Processing & Documentation **File:** `story-pdf-pptx-support-4.md` **Focus:** Comprehensive testing and integration **Key Tasks:** - Create test_pdf_conversion.py (unit tests) - Create test_pptx_conversion.py (unit tests) - Create test_pdf_preprocessing.py (image captioning tests) - Create test_pptx_preprocessing.py (preprocessing tests) - Create test_pdf_pptx_comprehensive.py (integration tests) - Update batch processing glob patterns (.pdf/.pptx/.ppt/.pptm) - Success rate validation (≥85%) - Error categorization for format-specific issues - Update README with PDF/PowerPoint support - Update extraction-pipeline-v2-tech-spec.md - Create troubleshooting guide **Acceptance:** Test suite passes, batch processor tracks PDF/PowerPoint, ≥85% success rate, docs updated --- ## Implementation Sequence ``` Story 1 (PDF Processing) Story 2 (PowerPoint Preprocessing) ↓ ↓ └───────────────┬────────────────────┘ ↓ Story 3 (Pipeline Integration) ↓ Story 4 (Testing & Documentation) ↓ Validation: Run full test suite, verify 85% success rate ``` **Dependencies:** - Story 1 and Story 2 can run in parallel (independent) - Story 3 depends on Story 1 and Story 2 (needs both converters) - Story 4 depends on Story 3 (needs complete pipeline) --- ## Technical Context **From Tech-Spec:** - PDF: All types supported (text-based, scanned, mixed) - PowerPoint: .pptx, .ppt, .pptm formats - Target: ≥85% extraction success rate - Follow Word document pattern (Markdown conversion + semantic chunking) **Stack:** - Python 3.11+, FastAPI 0.122, Celery 5.3+ - Docling 2.63.0+ (PDF → Markdown, PowerPoint → Markdown) - docling-core (HybridChunker, ImgTableAnnotationSerializerProvider) - tiktoken (OpenAI tokenizer for text-embedding-3-large) - python-pptx (PowerPoint speaker notes extraction) - NEW DEPENDENCY - LibreOffice 7.0+ CLI (format conversion) - PostgreSQL, Milvus 2.4+ **Key Files:** - `src/extraction_v2/document_converter.py` (MODIFY) - `src/extraction_v2/pptx_preprocessor.py` (CREATE - speaker notes only) - `src/extraction_v2/pipeline.py` (MODIFY) - `src/extraction_v2/semantic_chunker.py` (REUSE - existing with ImgTableAnnotationSerializerProvider) - `src/api/routes/documents.py` (MODIFY - file validation) - `src/workers/extraction_v2_tasks.py` (MODIFY) - `batch_extract_v2.py` (MODIFY - if exists) --- ## Epic Value **Business Impact:** - Support common enterprise document formats (PDF, PowerPoint) - Complete document coverage for production readiness - Enable processing of presentations and reports - Extract valuable structured data from slides and PDFs - Competitive parity with comprehensive document processing systems **Developer Impact:** - Reusable preprocessing patterns across document types - Unified Markdown conversion strategy (Word, PDF, PowerPoint) - Consistent semantic chunking approach - Comprehensive test coverage - Clear troubleshooting guides --- ## Epic Notes - **Unified Strategy:** All text documents (Word, PDF, PowerPoint) → Markdown → HybridChunker with custom serializers - **Excel Exception:** Excel remains HTML-based for table structure preservation - **LibreOffice Pattern:** Reuses .doc/.docm conversion approach for .ppt/.pptm - **Image & Table Handling:** Custom serializers (ImgTableAnnotationSerializerProvider) handle images and tables during chunking - NO preprocessing needed - **Simpler Approach:** Docling handles all image/table extraction automatically via custom serializers - **Docling Power:** Handles PDF OCR automatically, PowerPoint slide structure natively, images/tables via serializers - **Speaker Notes:** PowerPoint-specific feature, embedded inline via PptxPreprocessor as `[SPEAKER NOTES]: {text}` - **Token Limits:** 8000 tokens max per chunk (text-embedding-3-large model) - **No Image Captioning:** Unlike Word documents, PDF/PowerPoint rely on Docling's native image handling through serializers --- ## Risk Mitigation **Risk 1: LibreOffice Conversion Failures** - Mitigation: 60-second timeout, retry logic, file locking, graceful fallback **Risk 2: PDF Complexity (Scanned, Mixed)** - Mitigation: Docling handles OCR internally, validate with diverse test files **Risk 3: Image/Table Serialization** - Mitigation: Use proven ImgTableAnnotationSerializerProvider from Word pipeline, extensive testing **Risk 4: Speaker Notes Extraction** - Mitigation: Use python-pptx library, well-documented API, extensive testing --- **Epic Status:** Ready for Sprint Planning **Estimated Stories:** 4 stories **Story Size:** Medium (2-5 days each depending on complexity)