# Epic 3: Celery Deprecation & Cleanup - Stories

**Epic:** Celery Deprecation & Cleanup
**Phase:** 3
**Prerequisite:** Epic 2 Complete (100% traffic on Airflow, stable for 1 week)

---

## Story 3.1: Verify 100% Airflow Stability (1 Week Soak)

**Priority:** High | **Estimate:** Ongoing (1 week) | **Status:** pending

### User Story
As a **DevOps engineer**, I want to **verify Airflow stability over 1 week** so that **we can confidently deprecate Celery**.

### Acceptance Criteria
- [ ] 7 days of 100% Airflow traffic
- [ ] Success rate ≥95% maintained
- [ ] No critical incidents
- [ ] Processing time within SLA
- [ ] No manual interventions required

### Monitoring Checklist (Daily)
- [ ] Success rate check
- [ ] Error rate check
- [ ] Processing time P50/P95/P99
- [ ] Queue depth (if applicable)
- [ ] Resource utilization

### Tasks
- [ ] Day 1: Initial metrics baseline
- [ ] Day 2-3: Monitor and document any issues
- [ ] Day 4-5: Address any minor issues found
- [ ] Day 6-7: Final stability confirmation
- [ ] Sign-off document for deprecation

---

## Story 3.2: Remove Celery Extraction Task

**Priority:** Medium | **Estimate:** 2 hours | **Status:** pending

### User Story
As a **developer**, I want to **remove the Celery extraction task** so that **we eliminate the deprecated code path**.

### Acceptance Criteria
- [ ] `extract_document_v2` task removed
- [ ] Celery app no longer registers extraction task
- [ ] API no longer references Celery for extraction
- [ ] Feature flag infrastructure removed (no longer needed)
- [ ] Tests updated to remove Celery references

### Files to Modify/Delete

| File | Action |
|------|--------|
| `src/workers/extraction_v2_tasks.py` | Delete or archive |
| `src/workers/celery_app.py` | Remove extraction task import |
| `src/api/routes/documents.py` | Remove Celery path |
| `src/services/feature_flags.py` | Remove or keep for other uses |
| `tests/test_extraction_v2_tasks.py` | Delete |

### Tasks
- [ ] Remove `extract_document_v2` from Celery app
- [ ] Update API to only use Airflow
- [ ] Remove feature flag logic
- [ ] Delete extraction task tests
- [ ] Verify no broken imports

---

## Story 3.3: Archive Legacy Code

**Priority:** Low | **Estimate:** 1 hour | **Status:** pending

### User Story
As a **developer**, I want to **archive the legacy Celery code** so that **it's preserved for reference but not active in the codebase**.

### Acceptance Criteria
- [ ] Legacy code moved to archive folder
- [ ] Archive documented with reason and date
- [ ] Git history preserved
- [ ] No active references to archived code

### Archive Structure
```
archive/
└── celery-extraction-2025-12/
    ├── README.md (explanation)
    ├── extraction_v2_tasks.py
    └── tests/
        └── test_extraction_v2_tasks.py
```

### Tasks
- [ ] Create archive directory
- [ ] Move `extraction_v2_tasks.py` to archive
- [ ] Move related tests to archive
- [ ] Create README explaining archive
- [ ] Verify codebase compiles without archived files

---

## Story 3.4: Update API & Documentation

**Priority:** Medium | **Estimate:** 3 hours | **Status:** pending

### User Story
As a **developer**, I want to **update all documentation** so that **it reflects the new Airflow-based architecture**.

### Acceptance Criteria
- [ ] API documentation updated (OpenAPI/Swagger)
- [ ] Architecture documentation updated
- [ ] README updated with new processing flow
- [ ] Deployment documentation updated
- [ ] Troubleshooting guide updated

### Documentation Updates

| Document | Changes |
|----------|---------|
| `docs/architecture.md` | Add Airflow section, update diagrams |
| `docs/specs/extraction-pipeline-v2-tech-spec.md` | Mark as superseded |
| `README.md` | Update processing description |
| `docs/DEPLOYMENT.md` | Add Airflow deployment steps |
| `docs/TROUBLESHOOTING.md` | Add Airflow debugging guide |

### Tasks
- [ ] Update architecture documentation
- [ ] Update API documentation
- [ ] Update deployment guide
- [ ] Create Airflow troubleshooting guide
- [ ] Review and update README

---

## Story 3.5: Final Cleanup & Validation

**Priority:** Medium | **Estimate:** 2 hours | **Status:** pending

### User Story
As a **developer**, I want to **perform final cleanup and validation** so that **the migration is complete and documented**.

### Acceptance Criteria
- [ ] No Celery extraction code in active codebase
- [ ] All tests pass
- [ ] No unused imports or dependencies
- [ ] Migration documented in CHANGELOG
- [ ] Post-migration metrics documented

### Final Checklist
- [ ] `grep -r "extract_document_v2"` returns no active matches
- [ ] `grep -r "extraction_v2_tasks"` returns no active imports
- [ ] All unit tests pass
- [ ] All integration tests pass
- [ ] E2E tests pass with Airflow
- [ ] Docker build successful
- [ ] Deployment successful to staging

### Tasks
- [ ] Run codebase audit for legacy references
- [ ] Remove unused dependencies from requirements.txt
- [ ] Run full test suite
- [ ] Update CHANGELOG with migration note
- [ ] Create migration summary document
- [ ] Team sign-off

---

## Migration Summary Template

```markdown
# Airflow Migration Summary

**Completed:** YYYY-MM-DD
**Duration:** X weeks

## Key Metrics

| Metric | Before (Celery) | After (Airflow) |
|--------|-----------------|-----------------|
| Success Rate | X% | Y% |
| Avg Processing Time | Xs | Ys |
| P99 Processing Time | Xs | Ys |
| Manual Interventions/Week | X | Y |

## Benefits Realized

- [ ] Visual workflow management
- [ ] Task-level retries
- [ ] Better observability
- [ ] Reduced debugging time

## Lessons Learned

1. ...
2. ...

## Archive Location

Legacy Celery code archived at: `archive/celery-extraction-2025-12/`
```

---

## Dependencies

- Epic 2 complete (100% traffic stable)
- 1 week soak period completed
- Team sign-off for deprecation

## Risk Mitigation

Even after deprecation, keep:
- Archive of legacy code
- Ability to re-deploy Celery if critical issues found
- Database rollback capability (if schema changed)