# User Story: Testing, Batch Processing & Documentation

**Story ID:** story-pdf-pptx-support-4
**Epic:** PDF and PowerPoint Document Processing Support
**Created:** 2025-12-05
**Status:** Done
**Story Points:** 3
**Priority:** High

---

## User Story

**As a** data engineer
**I want** comprehensive testing, batch processing support, and complete documentation for PDF/PowerPoint processing
**So that** the feature is production-ready with ≥85% success rate and clear troubleshooting guides

---

## Context

**Current Situation:**
- Stories 1-3 complete: PDF and PowerPoint fully integrated into V2 pipeline
- Need comprehensive test coverage
- Need batch processing updates
- Need documentation updates

**Problem:**
Feature exists but lacks comprehensive testing, batch processing integration, and user-facing documentation.

**Solution:**
Create comprehensive test suite, update batch processor, create troubleshooting guides, update all relevant documentation.

---

## Technical Details

**Test Files to Create:**
- `tests/test_extraction_v2/test_pdf_conversion.py`
- `tests/test_extraction_v2/test_pptx_conversion.py`
- `tests/test_extraction_v2/test_pptx_preprocessing.py` (speaker notes only)
- `tests/test_extraction_v2/test_pdf_pptx_comprehensive.py`
- `tests/test_extraction_v2/test_pdf_pptx_serializers.py` (verify custom serializers)

**Documentation Files to Update:**
- `README.md`
- `docs/specs/extraction-pipeline-v2-tech-spec.md`
- `docs/feats/pdf_pptx_support/COMPLETION_SUMMARY.md` (CREATE)

**Batch Processing Updates:**
If batch_extract_v2.py exists:
```python
# Update glob patterns
file_patterns = [
    "**/*.xlsx",
    "**/*.xls",
    "**/*.xlsm",
    "**/*.docx",
    "**/*.doc",
    "**/*.docm",
    "**/*.pdf",        # NEW
    "**/*.pptx",       # NEW
    "**/*.ppt",        # NEW
    "**/*.pptm"        # NEW
]
```

**Comprehensive Test Suite:**
```python
# tests/test_extraction_v2/test_pdf_pptx_comprehensive.py

import pytest
from pathlib import Path
from src.extraction_v2.pipeline import ExtractionPipelineV2

class TestPdfPptxComprehensive:
    """Comprehensive end-to-end tests for PDF and PowerPoint support."""

    @pytest.fixture
    def pipeline(self):
        return ExtractionPipelineV2()

    def test_pdf_text_based(self, pipeline):
        """Test text-based PDF extraction."""
        result = pipeline.process_pdf_document_sync(
            document_id=uuid.uuid4(),
            file_path="tests/fixtures/text_based.pdf"
        )
        assert result.success is True
        assert result.record_count > 0
        assert all('content' in r for r in result.records)

    def test_pdf_scanned_with_ocr(self, pipeline):
        """Test scanned PDF with OCR."""
        result = pipeline.process_pdf_document_sync(
            document_id=uuid.uuid4(),
            file_path="tests/fixtures/scanned.pdf"
        )
        assert result.success is True
        # Docling should handle OCR automatically

    def test_pdf_mixed_content(self, pipeline):
        """Test mixed content PDF (text + scanned)."""
        result = pipeline.process_pdf_document_sync(
            document_id=uuid.uuid4(),
            file_path="tests/fixtures/mixed.pdf"
        )
        assert result.success is True

    def test_pdf_with_images(self, pipeline):
        """Test PDF with embedded images (handled by serializers)."""
        result = pipeline.process_pdf_document_sync(
            document_id=uuid.uuid4(),
            file_path="tests/fixtures/with_images.pdf"
        )
        assert result.success is True
        # Images handled by ImgTableAnnotationSerializerProvider during chunking
        # No metadata files needed

    def test_pptx_native(self, pipeline):
        """Test native .pptx file."""
        result = pipeline.process_pptx_document_sync(
            document_id=uuid.uuid4(),
            file_path="tests/fixtures/presentation.pptx"
        )
        assert result.success is True
        assert result.record_count > 0

    def test_ppt_conversion(self, pipeline):
        """Test .ppt legacy format conversion."""
        result = pipeline.process_pptx_document_sync(
            document_id=uuid.uuid4(),
            file_path="tests/fixtures/legacy.ppt"
        )
        assert result.success is True

    def test_pptm_macro_stripping(self, pipeline):
        """Test .pptm conversion with macro stripping."""
        result = pipeline.process_pptx_document_sync(
            document_id=uuid.uuid4(),
            file_path="tests/fixtures/macro_enabled.pptm"
        )
        assert result.success is True
        # Verify macros were stripped

    def test_pptx_with_speaker_notes(self, pipeline):
        """Test PowerPoint with speaker notes."""
        result = pipeline.process_pptx_document_sync(
            document_id=uuid.uuid4(),
            file_path="tests/fixtures/with_notes.pptx"
        )
        assert result.success is True
        # Check that speaker notes are in content
        has_notes = any('[SPEAKER NOTES]' in r['content'] for r in result.records)
        assert has_notes

    def test_success_rate_validation(self, pipeline):
        """Validate ≥85% success rate across all test files."""
        test_files = list(Path("tests/fixtures").glob("*.pdf")) + \
                     list(Path("tests/fixtures").glob("*.ppt*"))
        
        successes = 0
        for file_path in test_files:
            try:
                if file_path.suffix == '.pdf':
                    result = pipeline.process_pdf_document_sync(uuid.uuid4(), str(file_path))
                else:
                    result = pipeline.process_pptx_document_sync(uuid.uuid4(), str(file_path))
                
                if result.success:
                    successes += 1
            except Exception as e:
                print(f"Failed: {file_path.name} - {e}")
        
        success_rate = (successes / len(test_files)) * 100
        print(f"Success rate: {success_rate:.1f}%")
        assert success_rate >= 85.0
```

**Documentation Updates:**
```markdown
# README.md additions

## Supported Document Formats

DSOL V2 extraction pipeline supports the following document formats:

### Excel Files
- .xlsx (Office Open XML)
- .xls (legacy Excel 97-2003)
- .xlsm (macro-enabled)

### Word Documents
- .docx (Office Open XML)
- .doc (legacy Word 97-2003)
- .docm (macro-enabled)

### PDF Documents ✨ NEW
- Text-based PDFs
- Scanned PDFs (OCR handled automatically)
- Mixed-content PDFs
- PDFs with embedded images

### PowerPoint Presentations ✨ NEW
- .pptx (Office Open XML)
- .ppt (legacy PowerPoint 97-2003)
- .pptm (macro-enabled)

## Unified Processing Strategy

- **Excel:** → HTML → Table extraction → Structured records
- **Word/PDF/PowerPoint:** → Markdown → Semantic chunking → Contextualized chunks

All text documents (Word, PDF, PowerPoint) use Docling HybridChunker with OpenAI tokenizer for consistent semantic chunking (8000 tokens max).
```

---

## Implementation Steps

1. **Create comprehensive test suite**
   - Create all test files listed above
   - Add test fixtures (sample PDFs, PowerPoints)
   - Implement ≥85% success rate validation

2. **Update batch processor (if exists)**
   - Add PDF and PowerPoint to glob patterns
   - Update status tracking
   - Add format-specific error categorization

3. **Update documentation**
   - Update README.md with new formats
   - Update extraction-pipeline-v2-tech-spec.md
   - Create troubleshooting guide

4. **Create completion summary**
   - Document what was implemented
   - List all new files created
   - Capture metrics and success rates

5. **Run full validation**
   ```bash
   # Run all tests
   uv run pytest tests/test_extraction_v2/test_pdf*.py -v
   uv run pytest tests/test_extraction_v2/test_pptx*.py -v
   uv run pytest tests/test_extraction_v2/test_pdf_pptx_comprehensive.py -v
   
   # Validate success rate
   uv run pytest tests/test_extraction_v2/test_pdf_pptx_comprehensive.py::test_success_rate_validation -v
   ```

---

## Acceptance Criteria

1. ✅ Comprehensive test suite created
   - Unit tests for PDF conversion
   - Unit tests for PowerPoint conversion
   - Unit tests for preprocessing
   - Integration tests for end-to-end pipeline
   - Success rate validation test

2. ✅ All tests passing
   - 100% of unit tests pass
   - ≥85% success rate on comprehensive test suite
   - No regressions in existing Excel/Word tests

3. ✅ Batch processing updated (if applicable)
   - PDF and PowerPoint included in glob patterns
   - Status tracking functional
   - Metrics reporting working

4. ✅ Documentation complete
   - README updated with PDF/PowerPoint support
   - Tech-spec updated
   - Troubleshooting guide created
   - Completion summary documented

5. ✅ Error handling validated
   - All error scenarios tested
   - Clear error messages
   - Graceful fallbacks

6. ✅ Performance validated
   - PDF processing: 5-20 seconds per document
   - PowerPoint processing: 10-30 seconds per document
   - Image captioning: 1-3 seconds per image

---

## Testing Strategy

**Test Coverage Goals:**
- Unit test coverage: ≥90%
- Integration test coverage: ≥85%
- Success rate: ≥85%

**Test Categories:**
1. **Unit Tests:**
   - PDF conversion
   - PowerPoint conversion
   - Speaker notes preprocessing (PowerPoint only)
   - Format normalization
   - File validation

2. **Integration Tests:**
   - End-to-end PDF pipeline
   - End-to-end PowerPoint pipeline
   - Custom serializers (images/tables)
   - Batch processing
   - Error scenarios

3. **Success Rate Tests:**
   - Diverse document collection
   - Various PDF types
   - Various PowerPoint formats
   - Edge cases

---

## Dependencies

**Internal Dependencies:**
- Stories 1-3 must be complete
- Test fixtures need to be created

**External Dependencies:**
- pytest 8.0+
- pytest-asyncio 0.23+

**Blockers:**
- Requires Stories 1, 2, and 3 completion

---

## Definition of Done

- [ ] All test files created
- [ ] Test fixtures prepared
- [ ] Comprehensive test suite passing
- [ ] Success rate ≥85% validated
- [ ] Batch processor updated (if applicable)
- [ ] README.md updated
- [ ] Tech-spec updated
- [ ] Troubleshooting guide created
- [ ] COMPLETION_SUMMARY.md created
- [ ] All documentation reviewed
- [ ] Feature marked as production-ready
- [ ] Code reviewed and merged

---

## Reference

**Tech-Spec:** `docs/feats/pdf_pptx_support/tech-spec.md`
**Epic:** `docs/feats/pdf_pptx_support/epics.md`
**Acceptance Criteria:** Tech-spec section

---

**Story Status:** Ready for Development (depends on Stories 1, 2, 3)
