# User Story: PowerPoint Markdown Conversion & Pipeline Integration

**Story ID:** story-pdf-pptx-support-3
**Epic:** PDF and PowerPoint Document Processing Support
**Created:** 2025-12-05
**Status:** Done
**Story Points:** 3
**Priority:** High

---

## User Story

**As a** data engineer
**I want** PDF and PowerPoint documents fully integrated into the V2 pipeline with Markdown output and semantic chunking
**So that** all document types follow a unified processing pattern with consistent chunk records using custom serializers

---

## Context

**Current Situation:**
- Story 1 completed: PDF conversion (Docling direct)
- Story 2 completed: PowerPoint conversion and speaker notes preprocessing
- Need to integrate both into extraction_v2/pipeline.py
- Need to route through semantic chunker with ImgTableAnnotationSerializerProvider

**Problem:**
PDF and PowerPoint conversions exist but aren't integrated into the main extraction pipeline.

**Solution:**
Add process_pdf_document() and process_pptx_document() methods to pipeline.py, integrate with SemanticChunker using ImgTableAnnotationSerializerProvider for automatic image/table handling.

---

## Technical Details

**Files to Modify:**
- `src/extraction_v2/pipeline.py` - Add PDF/PowerPoint processing methods
- `src/extraction_v2/document_converter.py` - Add _convert_pptx_to_markdown() method
- `src/workers/extraction_v2_tasks.py` - Add PDF/PowerPoint file type handling

**Pipeline Integration:**
```python
# In pipeline.py

async def process_pdf_document(
    self,
    document_id: UUID,
    file_path: str
) -> ExtractionResult:
    """
    Process PDF document with semantic chunking.
    
    Images and tables handled by ImgTableAnnotationSerializerProvider.
    """
    try:
        logger.info(f"Starting V2 PDF extraction for {document_id}")

        # Step 1: PDF → Markdown (via Docling)
        markdown = self.converter.convert_to_html(file_path)  # Returns Markdown for PDF
        if markdown is None:
            return ExtractionResult(
                success=False,
                error="PDF conversion failed"
            )

        # Step 2: Get DoclingDocument for chunking
        from docling.document_converter import DocumentConverter, PdfFormatOption
        from docling.datamodel.base_models import InputFormat

        converter = DocumentConverter(
            format_options={InputFormat.PDF: PdfFormatOption()}
        )
        _ensure_event_loop()
        result = converter.convert(file_path)
        doc = result.document

        # Step 3: Chunk with custom serializers
        # SemanticChunker uses ImgTableAnnotationSerializerProvider automatically
        chunks = self.semantic_chunker.chunk_document(
            doc=doc,
            file_id=document_id,
            original_filename=Path(file_path).name
        )

        logger.info(f"V2 PDF pipeline complete: {len(chunks)} chunks")

        return ExtractionResult(
            success=True,
            document_id=document_id,
            file_path=file_path,
            record_count=len(chunks),
            records=chunks
        )

    except Exception as e:
        logger.error(f"V2 PDF pipeline failed: {e}", exc_info=True)
        return ExtractionResult(success=False, error=str(e))


async def process_pptx_document(
    self,
    document_id: UUID,
    file_path: str
) -> ExtractionResult:
    """
    Process PowerPoint document with semantic chunking.
    
    Speaker notes embedded during preprocessing.
    Images and tables handled by ImgTableAnnotationSerializerProvider during chunking.
    """
    try:
        logger.info(f"Starting V2 PowerPoint extraction for {document_id}")

        # Step 1: PowerPoint → Markdown (via Docling)
        # This includes speaker notes if preprocessed
        markdown = self.converter.convert_to_html(file_path)  # Returns Markdown for PowerPoint
        if markdown is None:
            return ExtractionResult(
                success=False,
                error="PowerPoint conversion failed"
            )

        # Step 2: Get DoclingDocument for chunking
        from pathlib import Path
        from docling.document_converter import DocumentConverter, PptxFormatOption
        from docling.datamodel.base_models import InputFormat

        file_ext = Path(file_path).suffix.lower()
        
        # Use preprocessed file if available
        preprocessed_path = Path(file_path).parent / f"{Path(file_path).stem}_preprocessed.pptx"
        if preprocessed_path.exists():
            conversion_path = str(preprocessed_path)
        else:
            conversion_path = file_path

        converter = DocumentConverter(
            format_options={InputFormat.PPTX: PptxFormatOption()}
        )
        _ensure_event_loop()
        result = converter.convert(conversion_path)
        doc = result.document

        # Step 3: Chunk with custom serializers
        # SemanticChunker uses ImgTableAnnotationSerializerProvider automatically
        chunks = self.semantic_chunker.chunk_document(
            doc=doc,
            file_id=document_id,
            original_filename=Path(file_path).name
        )

        logger.info(f"V2 PowerPoint pipeline complete: {len(chunks)} chunks")

        return ExtractionResult(
            success=True,
            document_id=document_id,
            file_path=file_path,
            record_count=len(chunks),
            records=chunks
        )

    except Exception as e:
        logger.error(f"V2 PowerPoint pipeline failed: {e}", exc_info=True)
        return ExtractionResult(success=False, error=str(e))


# Sync versions
def process_pdf_document_sync(self, document_id: UUID, file_path: str) -> ExtractionResult:
    """Synchronous version of process_pdf_document."""
    import asyncio
    import concurrent.futures

    def run_async_in_thread():
        new_loop = asyncio.new_event_loop()
        asyncio.set_event_loop(new_loop)
        try:
            return new_loop.run_until_complete(
                self.process_pdf_document(document_id, file_path)
            )
        finally:
            pending = asyncio.all_tasks(new_loop)
            for task in pending:
                task.cancel()
            if pending:
                new_loop.run_until_complete(asyncio.gather(*pending, return_exceptions=True))
            new_loop.close()

    with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
        future = executor.submit(run_async_in_thread)
        return future.result()


def process_pptx_document_sync(self, document_id: UUID, file_path: str) -> ExtractionResult:
    """Synchronous version of process_pptx_document."""
    import asyncio
    import concurrent.futures

    def run_async_in_thread():
        new_loop = asyncio.new_event_loop()
        asyncio.set_event_loop(new_loop)
        try:
            return new_loop.run_until_complete(
                self.process_pptx_document(document_id, file_path)
            )
        finally:
            pending = asyncio.all_tasks(new_loop)
            for task in pending:
                task.cancel()
            if pending:
                new_loop.run_until_complete(asyncio.gather(*pending, return_exceptions=True))
            new_loop.close()

    with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
        future = executor.submit(run_async_in_thread)
        return future.result()
```

**Document Converter - Add PowerPoint Markdown Conversion:**
```python
# In document_converter.py

def _convert_pptx_to_markdown(self, file_path: str) -> Optional[str]:
    """
    Convert PowerPoint to Markdown.

    Pipeline:
    1. Format normalization (.ppt/.pptm → .pptx if needed)
    2. Preprocess: Extract speaker notes (via PptxPreprocessor)
    3. Convert to Markdown (via Docling)
    
    Images/tables handled by custom serializers during chunking.
    """
    file_path_obj = Path(file_path)
    temp_pptx_path = None
    preprocessed_path = None

    try:
        file_ext = file_path_obj.suffix.lower()

        # Step 1: Format normalization
        if file_ext in ['.ppt', '.pptm']:
            logger.info(f"Converting {file_ext} to .pptx via LibreOffice...")
            temp_pptx_path = self._convert_ppt_to_pptx(file_path_obj)

            if temp_pptx_path is None:
                logger.error("LibreOffice conversion failed")
                return None

            file_to_preprocess = Path(temp_pptx_path)
        else:
            file_to_preprocess = file_path_obj

        # Step 2: Preprocessing (speaker notes only)
        try:
            from src.extraction_v2.pptx_preprocessor import PptxPreprocessor

            preprocessor = PptxPreprocessor()
            preprocessed_path = preprocessor.preprocess(file_to_preprocess)

            logger.info("PowerPoint speaker notes preprocessing complete")
        except Exception as preprocess_error:
            logger.warning(f"PowerPoint preprocessing failed: {preprocess_error}, proceeding without")
            preprocessed_path = file_to_preprocess

        # Step 3: Convert to Markdown via Docling
        logger.info(f"Converting PowerPoint to Markdown via Docling: {preprocessed_path}")
        _ensure_event_loop()
        result = self.converter.convert(str(preprocessed_path))

        if not result or not result.document:
            raise RuntimeError("Docling conversion returned empty result")

        markdown = result.document.export_to_markdown()
        logger.info(f"Successfully converted PowerPoint to Markdown ({len(markdown)} chars)")
        return markdown

    except Exception as e:
        logger.error(f"Docling PowerPoint conversion failed: {e}")
        return None

    finally:
        # Clean up temporary files
        for temp_file in [temp_pptx_path, preprocessed_path]:
            if temp_file and temp_file != file_path_obj and Path(temp_file).exists():
                try:
                    os.remove(temp_file)
                except Exception as e:
                    logger.warning(f"Failed to remove temp file {temp_file}: {e}")


def convert_to_html(self, file_path: str) -> Optional[str]:
    """
    Convert document to HTML/Markdown.
    
    Routes:
    - Excel → HTML
    - Word → Markdown
    - PDF → Markdown
    - PowerPoint → Markdown
    """
    file_ext = Path(file_path).suffix.lower()

    if file_ext in ['.xlsx', '.xls', '.xlsm']:
        return self._convert_excel_to_html(file_path)
    elif file_ext in ['.docx', '.doc', '.docm']:
        return self._convert_word_to_markdown(file_path)
    elif file_ext == '.pdf':
        return self._convert_pdf_to_markdown(file_path)
    elif file_ext in ['.pptx', '.ppt', '.pptm']:
        return self._convert_pptx_to_markdown(file_path)
    else:
        logger.error(f"Unsupported file format: {file_ext}")
        return None
```

---

## Implementation Steps

1. **Add PDF processing methods to pipeline.py**
   - Implement process_pdf_document() async method
   - Implement process_pdf_document_sync() method
   - Integrate with SemanticChunker

2. **Add PowerPoint processing methods to pipeline.py**
   - Implement process_pptx_document() async method
   - Implement process_pptx_document_sync() method
   - Handle preprocessed file detection

3. **Add _convert_pptx_to_markdown() to document_converter.py**
   - Format normalization (.ppt/.pptm → .pptx)
   - Speaker notes preprocessing
   - Docling conversion

4. **Update convert_to_html() routing**
   - Add PowerPoint to format detection
   - Route to _convert_pptx_to_markdown()

5. **Update Celery worker**
   - Modify src/workers/extraction_v2_tasks.py
   - Add PDF and PowerPoint to file type handling
   - Update error categorization

6. **Test end-to-end pipeline**
   ```bash
   # Test PDF pipeline
   python -c "
   from src.extraction_v2.pipeline import ExtractionPipelineV2
   import uuid
   pipeline = ExtractionPipelineV2()
   result = pipeline.process_pdf_document_sync(uuid.uuid4(), 'test.pdf')
   assert result.success
   assert len(result.records) > 0
   "

   # Test PowerPoint pipeline
   python -c "
   from src.extraction_v2.pipeline import ExtractionPipelineV2
   import uuid
   pipeline = ExtractionPipelineV2()
   result = pipeline.process_pptx_document_sync(uuid.uuid4(), 'test.pptx')
   assert result.success
   assert len(result.records) > 0
   "
   ```

---

## Acceptance Criteria

1. ✅ PDF processing methods added to pipeline
   - process_pdf_document() async method working
   - process_pdf_document_sync() method working
   - Chunks generated correctly

2. ✅ PowerPoint processing methods added to pipeline
   - process_pptx_document() async method working
   - process_pptx_document_sync() method working
   - Speaker notes preserved in chunks

3. ✅ _convert_pptx_to_markdown() implemented
   - Format normalization working
   - Speaker notes preprocessing integrated
   - Docling conversion functional

4. ✅ Document converter routing updated
   - PDF files route to _convert_pdf_to_markdown()
   - PowerPoint files route to _convert_pptx_to_markdown()
   - All formats supported

5. ✅ Celery worker updated
   - PDF and PowerPoint file types handled
   - Error categorization updated
   - Task routing functional

6. ✅ End-to-end pipeline working
   - PDF: File → Markdown → Chunking → Records (images via serializers)
   - PowerPoint: File → Speaker Notes → Markdown → Chunking → Records (images via serializers)

7. ✅ Custom serializers working
   - ImgTableAnnotationSerializerProvider handles images/tables
   - No separate image preprocessing needed
   - Consistent with Word pipeline

---

## Testing Strategy

**Integration Tests:**
```python
# tests/test_extraction_v2/test_pdf_pptx_pipeline.py

@pytest.mark.asyncio
async def test_pdf_end_to_end_pipeline():
    pipeline = ExtractionPipelineV2()
    doc_id = uuid.uuid4()
    
    result = await pipeline.process_pdf_document(doc_id, "test.pdf")
    
    assert result.success is True
    assert result.record_count > 0
    assert all('content' in r for r in result.records)
    # Verify images handled by serializers (not preprocessing)

@pytest.mark.asyncio
async def test_pptx_end_to_end_pipeline():
    pipeline = ExtractionPipelineV2()
    doc_id = uuid.uuid4()
    
    result = await pipeline.process_pptx_document(doc_id, "test.pptx")
    
    assert result.success is True
    assert result.record_count > 0
    assert all('metadata' in r for r in result.records)
    # Verify images handled by serializers

def test_pptx_with_speaker_notes():
    pipeline = ExtractionPipelineV2()
    result = pipeline.process_pptx_document_sync(uuid.uuid4(), "test_with_notes.pptx")
    
    assert result.success
    # Check that speaker notes appear in chunks
    has_notes = any('[SPEAKER NOTES]' in r['content'] for r in result.records)
    assert has_notes

def test_sync_pdf_pipeline():
    pipeline = ExtractionPipelineV2()
    result = pipeline.process_pdf_document_sync(uuid.uuid4(), "test.pdf")
    assert result.success

def test_sync_pptx_pipeline():
    pipeline = ExtractionPipelineV2()
    result = pipeline.process_pptx_document_sync(uuid.uuid4(), "test.pptx")
    assert result.success
```

---

## Dependencies

**Internal Dependencies:**
- Story 1 (PDF conversion) must be complete
- Story 2 (PowerPoint speaker notes preprocessing) must be complete
- SemanticChunker with ImgTableAnnotationSerializerProvider (existing)
- DocumentConverter updates

**External Dependencies:**
- Docling 2.63.0+ (PDF and PowerPoint support)
- docling-core (HybridChunker, ImgTableAnnotationSerializerProvider)

**Blockers:**
- Requires Story 1 and Story 2 completion

---

## Definition of Done

- [ ] process_pdf_document() implemented (async + sync)
- [ ] process_pptx_document() implemented (async + sync)
- [ ] _convert_pptx_to_markdown() implemented
- [ ] Document converter routing updated
- [ ] Celery worker updated for PDF/PowerPoint
- [ ] End-to-end integration tests passing
- [ ] PDF pipeline functional
- [ ] PowerPoint pipeline functional
- [ ] Speaker notes preserved in chunks
- [ ] Images/tables handled by serializers (verified)
- [ ] Chunk records properly formatted
- [ ] Metadata preservation working
- [ ] Error handling comprehensive
- [ ] Code reviewed
- [ ] Ready for testing and batch processing (Story 4)

---

## Reference

**Tech-Spec:** `docs/feats/pdf_pptx_support/tech-spec.md`
**Epic:** `docs/feats/pdf_pptx_support/epics.md`
**Pattern:** Word document pipeline (process_word_document with serializers)

---

**Story Status:** Ready for Development (depends on Story 1 & 2)
