# User Story: Create Comprehensive Test Suite

**Story ID:** story-v2-pipeline-testing-2
**Epic:** V2 Pipeline Comprehensive Testing
**Created:** 2025-12-01
**Status:** Ready for Development
**Story Points:** 5
**Priority:** High

---

## User Story

**As a** QA engineer
**I want** an automated test suite that validates extraction across all 58 CustomerDocument/ files
**So that** we can verify ≥95% success rate and catch regressions automatically

---

## Context

**Current Situation:**
- Manual testing only covers sample cases
- No automated regression testing against full dataset
- Success rate unknown for complete file set
- No parametrized tests for all CustomerDocument/ files

**Problem:**
Cannot confidently deploy V2 pipeline without comprehensive automated validation.

**Solution:**
Create pytest test suite that processes all 58 files, tracks metrics, and validates success criteria.

---

## Technical Details

**New Test File:**
`tests/test_extraction_v2/test_comprehensive_validation.py`

**Test Architecture:**
```python
import pytest
from pathlib import Path
from typing import Dict, List
from src.extraction_v2.pipeline import ExtractionPipelineV2
from src.extraction.pipeline import ExtractionPipeline  # V1 fallback
import structlog

logger = structlog.get_logger()

# Test fixtures
@pytest.fixture
def customer_document_files():
    """Discover all Excel files in CustomerDocument/."""
    folder = Path("CustomerDocument/")
    patterns = ["**/*.xlsx", "**/*.xls", "**/*.xlsm"]

    files = []
    for pattern in patterns:
        files.extend(folder.glob(pattern))

    return sorted(files)

@pytest.fixture
def v2_pipeline():
    """V2 extraction pipeline instance."""
    return ExtractionPipelineV2()

@pytest.fixture
def v1_pipeline():
    """V1 extraction pipeline for fallback."""
    return ExtractionPipeline()

# Parametrized test for all files
@pytest.mark.parametrize("file_path", [
    pytest.param(f, id=f.name)
    for f in sorted(Path("CustomerDocument/").rglob("*.xl*"))
])
async def test_extract_file_v2_with_fallback(
    file_path: Path,
    v2_pipeline,
    v1_pipeline,
    metrics_tracker
):
    """Test V2 extraction with V1 fallback for each file."""
    logger.info(f"Testing extraction", file=file_path.name)

    # Try V2 first
    try:
        result = await v2_pipeline.extract_from_file(str(file_path))

        if result and len(result) > 0:
            metrics_tracker.record_success('v2', file_path)
            assert True, f"V2 extraction successful: {len(result)} records"
            return
        else:
            logger.warning("V2 returned empty result", file=file_path.name)
            metrics_tracker.record_fallback(file_path, "Empty result")

    except Exception as e:
        logger.warning("V2 extraction failed", file=file_path.name, error=str(e))
        metrics_tracker.record_fallback(file_path, str(e))

    # Fallback to V1
    try:
        result_v1 = await v1_pipeline.extract_from_file(str(file_path))

        if result_v1 and len(result_v1) > 0:
            metrics_tracker.record_success('v1', file_path)
            assert True, f"V1 fallback successful: {len(result_v1)} records"
        else:
            metrics_tracker.record_failure(file_path, "Both V2 and V1 failed")
            pytest.fail(f"Both V2 and V1 extraction failed for {file_path.name}")

    except Exception as e:
        metrics_tracker.record_failure(file_path, f"V1 fallback error: {str(e)}")
        pytest.fail(f"V1 fallback failed: {str(e)}")
```

**Metrics Tracker:**
```python
class MetricsTracker:
    """Track extraction success metrics."""

    def __init__(self):
        self.v2_success: List[Path] = []
        self.v1_fallback: List[tuple[Path, str]] = []
        self.failed: List[tuple[Path, str]] = []

    def record_success(self, version: str, file_path: Path):
        if version == 'v2':
            self.v2_success.append(file_path)
        elif version == 'v1':
            self.v1_fallback.append((file_path, "V2 failed, V1 succeeded"))

    def record_fallback(self, file_path: Path, reason: str):
        self.v1_fallback.append((file_path, reason))

    def record_failure(self, file_path: Path, reason: str):
        self.failed.append((file_path, reason))

    def calculate_success_rate(self, total_files: int) -> float:
        """Calculate overall success rate (V2 + V1)."""
        total_success = len(self.v2_success) + len([f for f, _ in self.v1_fallback])
        return (total_success / total_files) * 100 if total_files > 0 else 0.0

    def generate_report(self) -> Dict:
        """Generate comprehensive metrics report."""
        total = len(self.v2_success) + len(self.v1_fallback) + len(self.failed)

        return {
            "total_files": total,
            "v2_success": len(self.v2_success),
            "v1_fallback": len(self.v1_fallback),
            "failed": len(self.failed),
            "success_rate": self.calculate_success_rate(total),
            "v2_success_rate": (len(self.v2_success) / total * 100) if total > 0 else 0.0,
            "failure_details": [
                {"file": f.name, "reason": r} for f, r in self.failed
            ],
            "fallback_details": [
                {"file": f.name, "reason": r} for f, r in self.v1_fallback
            ]
        }

@pytest.fixture(scope="session")
def metrics_tracker():
    """Session-scoped metrics tracker."""
    return MetricsTracker()
```

**Success Rate Validation:**
```python
def test_overall_success_rate(metrics_tracker, customer_document_files):
    """Validate that overall success rate meets 95% target."""
    total_files = len(customer_document_files)
    report = metrics_tracker.generate_report()

    logger.info("Comprehensive validation results", **report)

    # Assert 95% success rate
    assert report["success_rate"] >= 95.0, (
        f"Success rate {report['success_rate']:.1f}% is below 95% target. "
        f"Failed files: {report['failed']}"
    )

    # Assert all 58 files processed
    assert total_files == 58, f"Expected 58 files, found {total_files}"

    # Save report to file
    import json
    with open("validation_report.json", "w") as f:
        json.dump(report, f, indent=2, default=str)
```

---

## Implementation Steps

1. **Create test file structure**
   ```bash
   mkdir -p tests/test_extraction_v2
   touch tests/test_extraction_v2/test_comprehensive_validation.py
   touch tests/test_extraction_v2/conftest.py
   ```

2. **Implement MetricsTracker class**
   - Add to conftest.py as session-scoped fixture
   - Implement success/fallback/failure tracking
   - Add report generation logic

3. **Create parametrized test**
   - Use pytest.mark.parametrize with all 58 files
   - Implement V2 extraction with V1 fallback
   - Track results in MetricsTracker

4. **Add success rate validation test**
   - Run after all parametrized tests complete
   - Assert ≥95% success rate
   - Generate JSON report

5. **Update conftest.py fixtures**
   ```python
   # tests/test_extraction_v2/conftest.py
   import pytest
   from pathlib import Path

   @pytest.fixture(scope="session")
   def customer_document_files():
       """Discover all Excel files."""
       folder = Path("CustomerDocument/")
       patterns = ["**/*.xlsx", "**/*.xls", "**/*.xlsm"]

       files = []
       for pattern in patterns:
           files.extend(folder.glob(pattern))

       return sorted(files)

   @pytest.fixture
   def mock_azure_openai(monkeypatch):
       """Mock Azure OpenAI for faster tests."""
       # Mock LLM calls to avoid API costs during testing
       pass
   ```

6. **Run tests and verify**
   ```bash
   # Run comprehensive validation
   uv run pytest tests/test_extraction_v2/test_comprehensive_validation.py -v

   # Check success rate
   cat validation_report.json | jq '.success_rate'

   # View failure details
   cat validation_report.json | jq '.failure_details'
   ```

---

## Acceptance Criteria

1. ✅ Test suite processes all 58 files in CustomerDocument/
   - 22 .xlsx files tested
   - 15 .xls files tested
   - 21 .xlsm files tested

2. ✅ Parametrized test covers each file individually
   - Clear test IDs (file names)
   - V2 extraction attempted first
   - V1 fallback on V2 failure
   - Metrics tracked for each file

3. ✅ Success rate validation test implemented
   - Asserts ≥95% overall success rate
   - Asserts exactly 58 files processed
   - Generates validation_report.json

4. ✅ Metrics tracking comprehensive
   - V2 success count
   - V1 fallback count with reasons
   - Failure count with error details
   - Per-file-type statistics

5. ✅ Test output clear and actionable
   - Failed tests show file name and reason
   - Success rate displayed
   - Report saved to JSON for analysis

---

## Testing Strategy

**Unit Tests:**
```python
def test_metrics_tracker_success_rate():
    tracker = MetricsTracker()

    # Simulate 95 successes, 5 failures
    for i in range(90):
        tracker.record_success('v2', Path(f"file{i}.xlsx"))
    for i in range(5):
        tracker.record_success('v1', Path(f"file{i+90}.xlsx"))
    for i in range(5):
        tracker.record_failure(Path(f"file{i+95}.xlsx"), "Error")

    assert tracker.calculate_success_rate(100) == 95.0

def test_customer_document_files_fixture(customer_document_files):
    """Verify file discovery fixture."""
    assert len(customer_document_files) == 58

    xlsx_count = sum(1 for f in customer_document_files if f.suffix == ".xlsx")
    xls_count = sum(1 for f in customer_document_files if f.suffix == ".xls")
    xlsm_count = sum(1 for f in customer_document_files if f.suffix == ".xlsm")

    assert xlsx_count == 22
    assert xls_count == 15
    assert xlsm_count == 21
```

**Integration Test:**
```bash
# Full test suite run
uv run pytest tests/test_extraction_v2/test_comprehensive_validation.py \
    --tb=short \
    -v \
    --durations=10

# Verify report generated
test -f validation_report.json && echo "Report exists"

# Check success rate meets target
RATE=$(cat validation_report.json | jq '.success_rate')
echo "Success rate: $RATE%"
[ "$RATE" \> "95" ] && echo "✅ Target met" || echo "❌ Below target"
```

---

## Dependencies

**External Dependencies:**
- pytest 8.0+ (already installed)
- pytest-asyncio 0.23+ (async test support)
- structlog 24.0+ (logging)

**Internal Dependencies:**
- Story 1 must be complete (all 58 files discovered)
- ExtractionPipelineV2 implementation
- ExtractionPipeline (V1) for fallback
- CustomerDocument/ folder with 58 files

**Blockers:**
- Story 1 (file discovery) must be complete first
- Azure OpenAI credentials needed for LLM calls (or mock in tests)

---

## Definition of Done

- [ ] test_comprehensive_validation.py created with parametrized tests
- [ ] MetricsTracker class implemented and tested
- [ ] All 58 files processed by test suite
- [ ] Success rate validation test implemented
- [ ] validation_report.json generated automatically
- [ ] Test suite passes with ≥95% success rate
- [ ] conftest.py fixtures added
- [ ] Code committed to feature branch
- [ ] Documentation updated (README testing section)
- [ ] Ready for Story 3 (error handling uses these tests)

---

## Reference

**Tech-Spec:** `docs/feats/wrapup_pipeline_v2/tech-spec.md`
**Epic:** `docs/feats/wrapup_pipeline_v2/epics.md`
**Depends On:** Story 1 (file discovery)
**Key Files:**
- `tests/test_extraction_v2/test_comprehensive_validation.py` (create)
- `tests/test_extraction_v2/conftest.py` (modify)
- `CustomerDocument/` - 58 test files

---

**Story Status:** Ready for Development