---
id: DE-EPIC
title: "textiq-doc-extraction - Epic Breakdown"
version: "2.0"
status: draft
created: "2025-11-26"
lastUpdated: "2026-03-21"
author: nguyetnta8
document_type: epics
service: textiq-doc-extraction
---

# textiq-doc-extraction - Epic Breakdown

**Author:** TextIQ
**Date:** 2025-11-26
**Project Type:** api_backend
**Scale:** 119 Excel files, 33+ FRs

---

## Table of Contents

- [Overview](#overview)
- [Functional Requirements Inventory](#functional-requirements-inventory)
- [FR Coverage Map](#fr-coverage-map)
- [Epic 1: Foundation & API Skeleton](#epic-1-foundation--api-skeleton)
- [Epic 2: Document Upload & Management](#epic-2-document-upload--management)
- [Epic 3: Extraction Pipeline](#epic-3-extraction-pipeline)
- [Epic 4: Knowledge Base & Indexing](#epic-4-knowledge-base--indexing)
- [Epic 5: Q&A & Answer Generation](#epic-5-qa--answer-generation)
- [Epic 6: HITL Tagging & Tag Management (Phase 4)](#epic-6-hitl-tagging--tag-management-phase-4)
- [Epic 7: Dynamic Governance & Taxonomy Management (Phase 5)](#epic-7-dynamic-governance--taxonomy-management-phase-5)
- [Epic 8: LLM Provider Abstraction & Infrastructure](#epic-8-llm-provider-abstraction--infrastructure)
- [FR Coverage Matrix](#fr-coverage-matrix)
- [Summary](#summary)
- [Change History](#change-history)

---

## Overview

This document provides the complete epic and story breakdown for textiq-doc-extraction, decomposing the requirements from the [PRD](./prd.md) into implementable stories.

**Epics Summary:**

| Epic | Title | Stories | FRs Covered |
|------|-------|---------|-------------|
| 1 | Foundation & API Skeleton | 5 | Infrastructure for all FRs |
| 2 | Document Upload & Management | 6 | FR1-5, FR29-33 (partial) |
| 3 | Extraction Pipeline | 11 | FR6-17 |
| 4 | Knowledge Base & Indexing | 5 | FR18-21 |
| 5 | Q&A & Answer Generation | 6 | FR22-28, FR30-32 |
| 6 | HITL Tagging & Tag Management (Phase 4) | 5 | FR20-22 |
| 7 | Dynamic Governance & Taxonomy (Phase 5) | 4 | FR23-25 |
| 8 | LLM Provider Abstraction & Infrastructure | 3 | FR26, Infrastructure |

**Total:** 8 epics, 45 stories

---

## Functional Requirements Inventory

### Document Management
- **FR1:** System can ingest Excel files (.xlsx, .xls formats)
- **FR2:** System can process multiple files in a single batch upload
- **FR3:** System validates file format and reports errors for unsupported files
- **FR4:** System tracks processing status for each uploaded document
- **FR5:** Users can delete documents and their associated extracted data

### Table Extraction
- **FR6:** System detects table boundaries within Excel sheets
- **FR7:** System extracts multi-level headers preserving parent-child hierarchy
- **FR8:** System resolves merged cells by propagating values to covered cells
- **FR9:** System preserves row/column positional relationships
- **FR10:** System handles multiple tables per sheet

### Symbol Resolution
- **FR11:** System auto-detects sheets that function as symbol dictionaries
- **FR12:** System builds mappings from symbols to their meanings
- **FR13:** System resolves symbols in content before final indexing
- **FR14:** Users can upload custom symbol dictionaries

### Cross-Reference Handling
- **FR15:** System detects cross-reference patterns
- **FR16:** System resolves cross-references to actual content locations
- **FR17:** System maintains links between related content across sheets/files

### Knowledge Base
- **FR18:** System stores extracted content with full structural metadata
- **FR19:** System indexes content for exact text/code lookups
- **FR20:** System indexes content for semantic similarity search
- **FR21:** System maintains source location for all indexed content

### Q&A Processing
- **FR22:** Users can submit natural language questions via API
- **FR23:** System routes queries to appropriate retrieval method
- **FR24:** System generates answers using LLM with retrieved context
- **FR25:** System includes source references in every answer
- **FR26:** System provides confidence score with each answer
- **FR27:** System returns "not found" with explanation when answer unavailable
- **FR28:** Users can submit batch queries for multiple questions

### API Access
- **FR29:** System exposes REST API for all operations
- **FR30:** System authenticates requests via API key
- **FR31:** System rate-limits requests per API key
- **FR32:** System logs all requests and responses for debugging
- **FR33:** System provides health check endpoint

---

## FR Coverage Map

| FR | Epic | Story |
|----|------|-------|
| FR1 | Epic 2 | 2.2, 2.3 |
| FR2 | Epic 2 | 2.4 |
| FR3 | Epic 2 | 2.3 |
| FR4 | Epic 2 | 2.5 |
| FR5 | Epic 2 | 2.6 |
| FR6 | Epic 3 | 3.1 |
| FR7 | Epic 3 | 3.2 |
| FR8 | Epic 3 | 3.3 |
| FR9 | Epic 3 | 3.1, 3.4 |
| FR10 | Epic 3 | 3.1 |
| FR11 | Epic 3 | 3.5, 3.8a |
| FR12 | Epic 3 | 3.5, 3.8a |
| FR13 | Epic 3 | 3.6, 3.8a |
| FR14 | Epic 3 | 3.7 |
| FR15 | Epic 3 | 3.8, 3.8a |
| FR16 | Epic 3 | 3.8, 3.8a |
| FR17 | Epic 3 | 3.8, 3.8a |
| FR18 | Epic 4 | 4.1, 4.2 |
| FR19 | Epic 4 | 4.3 |
| FR20 | Epic 4, 6 | 4.4, 6.1, 6.2 |
| FR21 | Epic 4, 6 | 4.1, 4.2, 6.3 |
| FR22 | Epic 5 | 5.1 |
| FR23 | Epic 5 | 5.2 |
| FR24 | Epic 5 | 5.3 |
| FR25 | Epic 5 | 5.3 |
| FR26 | Epic 5 | 5.4 |
| FR27 | Epic 5 | 5.3 |
| FR28 | Epic 5 | 5.5 |
| FR29 | Epic 1 | 1.3 |
| FR30 | Epic 2 | 2.1 |
| FR31 | Epic 2 | 2.1 |
| FR32 | Epic 1 | 1.4 |
| FR33 | Epic 1 | 1.5 |

---

## Epic 1: Foundation & API Skeleton

**Goal:** Establish project infrastructure, development environment, and basic API framework that all subsequent features build upon.

**FRs Covered:** Infrastructure for all FRs, FR29, FR32, FR33

### Story 1.1: Project Initialization
### Story 1.2: Database Schema & Migrations
### Story 1.3: FastAPI Application Shell
### Story 1.4: Structured Logging Setup
### Story 1.5: Health Check Endpoint

_(Stories 1.1-1.5 retain original acceptance criteria and technical notes from impl repo. See impl `docs/epics.md` for full story details.)_

---

## Epic 2: Document Upload & Management

**Goal:** Enable clients to upload documents, track processing status, and manage their document library.

**FRs Covered:** FR1-5, FR29-33 (partial)

### Story 2.1: API Key Authentication
### Story 2.2: Single Document Upload
### Story 2.3: File Validation
### Story 2.4: Batch Document Upload
### Story 2.5: Document Status Tracking
### Story 2.6: Document Deletion

_(Stories 2.1-2.6 retain original acceptance criteria from impl repo.)_

---

## Epic 3: Extraction Pipeline

**Goal:** Build the multi-pass extraction engine that transforms documents into structured, semantically-rich records.

**FRs Covered:** FR6-17

### Story 3.1: Table Boundary Detection
### Story 3.1b: Horizontal Multi-Table Detection (Post-MVP)
### Story 3.2: Multi-Level Header Extraction
### Story 3.3: Merged Cell Resolution
### Story 3.4: Record Builder
### Story 3.4a: Extraction Pipeline Orchestrator
### Story 3.4b: Docling-Based Excel to HTML Converter
### Story 3.5: Symbol Dictionary Detection
### Story 3.6: Symbol Resolution in Content
### Story 3.7: Custom Symbol Dictionary Upload
### Story 3.8: Cross-Reference Detection and Resolution
### Story 3.8a: Multi-Pass Pipeline Integration

_(Stories 3.x retain original acceptance criteria from impl repo.)_

---

## Epic 4: Knowledge Base & Indexing

**Goal:** Store extracted content in structured and vector databases for efficient retrieval.

**FRs Covered:** FR18-21

### Story 4.1: Structured Store (PostgreSQL)
### Story 4.2: Source Metadata Preservation
### Story 4.3: Exact Lookup Indexing
### Story 4.4: Vector Embedding and Indexing
### Story 4.5: Incremental Indexing

_(Stories 4.x retain original acceptance criteria from impl repo.)_

---

## Epic 5: Q&A & Answer Generation

**Goal:** Enable clients to ask natural language questions and receive accurate, sourced answers.

**FRs Covered:** FR22-28, FR30-32

### Story 5.1: Query Endpoint
### Story 5.2: Query Router
### Story 5.3: Answer Generation with Sources
### Story 5.4: Confidence Scoring
### Story 5.5: Batch Query
### Story 5.6: Query Logging

_(Stories 5.x retain original acceptance criteria from impl repo.)_

---

## Epic 6: HITL Tagging & Tag Management (Phase 4)

> **C-REFRESH 2026-03-21:** New epic added based on code analysis of implemented tag services.

**Goal:** Enable human-in-the-loop tag management with validation, review workflows, and semantic tagging.

**User Value:** Users can add, review, and manage semantic tags on extracted records, improving retrieval quality through human oversight.

**FRs Covered:** FR20-22 (Tagging & Governance)

---

### Story 6.1: Tag Management Service

As a **system**,
I want **CRUD operations for semantic tags on extracted records**,
So that **users and AI can collaboratively manage tags**.

**Acceptance Criteria:**

**Given** an extracted record exists
**When** a user adds a tag via `POST /v1/blocks/{record_id}/tags`
**Then** the system:
1. Validates tag name format (`namespace:value`) via `TagValidator`
2. Checks against Standard Taxonomy via `TaxonomyLoader`
3. If tag is pre-defined: status = `verified`, source = `human`
4. If tag is dynamic (unknown namespace): status = `pending`, flagged for review
5. If tag is rejected (forbidden namespace): returns validation error
6. Stores tag as JSONB in `extracted_records.tags` array

**And** tag edit replaces existing tag by `replace_tag_id` parameter
**And** tag delete marks tag status as `deleted` with history entry

**Technical Notes:**
- Module: `src/services/tag_management_service.py`
- Schemas: `src/api/schemas/tags.py` (`TagCreate`, `TagCreateResponse`, `TagAssignment`, `TagStatus`)
- Route: `src/api/routes/tags.py`

---

### Story 6.2: Semantic Tagger Pipeline

As a **system**,
I want **LLM-based automatic tag classification during extraction**,
So that **records are tagged with semantic labels without manual intervention**.

**Acceptance Criteria:**

**Given** extracted content (records or hierarchical nodes)
**When** the `tag_records` or `tag_records_nodes` Airflow task runs
**Then** the `SemanticTagger`:
1. Builds dynamic prompts from taxonomy config via `PromptBuilder`
2. Sends content to LLM for classification
3. Assigns tags with confidence scores (namespace:value format)
4. Pre-defined tags auto-verified; dynamic tags flagged as pending
5. Tag inheritance applied from document -> section -> block via `InheritanceEngine`

**Technical Notes:**
- Module: `src/extraction_v2/semantic_tagger.py`
- Supports ISO-compliant taxonomy with Hierarchy of Intent (Phase 2)
- Tag inheritance via `src/extraction_v2/inheritance_engine.py`
- Pattern-based section inference: requirements, security, performance, testing, architecture, API, database, deployment, documentation, compliance

---

### Story 6.3: HITL Tag Review Workflow

As a **reviewer**,
I want **to confirm or reject pending tags**,
So that **only verified tags are used for retrieval**.

**Acceptance Criteria:**

**Given** a record has tags with status "pending"
**When** a reviewer calls `POST /v1/blocks/{record_id}/tags/review`
**Then** the system:
1. Finds the target tag by `tag_id`
2. Applies decision (`confirm` or `reject`)
3. Updates tag status to `verified` or `rejected`
4. Records history entry: reviewer, timestamp, decision, reason

**Technical Notes:**
- Module: `src/services/tag_review_service.py`
- Schemas: `TagReview`, `TagReviewResponse`, `TagReviewDecision`
- Route: `POST /v1/blocks/{record_id}/tags/review`

---

### Story 6.4: Tag Validation Engine

As a **system**,
I want **tags validated against the Standard Taxonomy before persistence**,
So that **tag quality is maintained**.

**Acceptance Criteria:**

**Given** a tag name in `namespace:value` format
**When** validation runs via `TagValidator`
**Then:**
- Pre-defined tags (in taxonomy): `is_valid=True`, `status=verified`
- Dynamic namespace tags (topic, entity, tech, domain): `is_valid=True`, `status=flagged`
- Forbidden namespace tags: `is_valid=False`, `status=rejected`

**Technical Notes:**
- Module: `src/extraction_v2/tag_validator.py`
- Taxonomy loaded by: `src/extraction_v2/taxonomy_loader.py` (supports profile: fintech, healthcare, automotive)

---

### Story 6.5: Tag API Endpoints

As an **API consumer**,
I want **REST endpoints for tag operations**,
So that **I can manage tags programmatically**.

**Endpoints:**
- `GET /v1/blocks/{record_id}/tags` -- List all tags with metadata
- `POST /v1/blocks/{record_id}/tags` -- Add/edit tag
- `POST /v1/blocks/{record_id}/tags/review` -- Review pending tag

**Technical Notes:**
- Module: `src/api/routes/tags.py`
- Router prefix: `/v1/blocks`

---

## Epic 7: Dynamic Governance & Taxonomy Management (Phase 5)

> **C-REFRESH 2026-03-21:** New epic added based on code analysis of governance services.

**Goal:** Enable admins to identify trending tags and promote them to the Standard Taxonomy.

**User Value:** The taxonomy evolves organically based on actual usage patterns, with admin oversight.

**FRs Covered:** FR23-25

---

### Story 7.1: Trending Tag Analysis

As an **admin**,
I want **to identify trending dynamic tags**,
So that **I can decide which tags to promote to the Standard Taxonomy**.

**Acceptance Criteria:**

**Given** records with dynamic tags exist
**When** admin calls `GET /v1/admin/taxonomy/trending`
**Then** the system returns trending tags with:
- `name`: tag name (e.g., "topic:gdpr")
- `usage_count`: total uses across all records
- `document_count`: number of documents using this tag
- `verified_ratio`: ratio of verified to total uses
- `first_seen` / `last_seen`: date range

**And** supports filters: `min_usage` (default 50), `min_documents` (default 3), `namespace`

**Technical Notes:**
- Module: `src/services/tag_governance_service.py` (`DynamicTagAnalyzer`)
- SQL aggregation from JSONB `tags` column with `jsonb_array_elements`

---

### Story 7.2: Tag Usage Statistics

As an **admin**,
I want **dashboard-level tag statistics**,
So that **I can understand tag distribution and quality**.

**Acceptance Criteria:**

**Given** records with tags exist
**When** admin calls `GET /v1/admin/taxonomy/stats`
**Then** the system returns:
- `total_tags`: total tag assignments
- `by_namespace`: count per namespace (req, topic, risk, info, etc.)
- `by_status`: count per status (verified, pending, rejected)
- `dynamic_tags`: count of dynamic namespace tags

**Technical Notes:**
- Module: `src/services/tag_governance_service.py` (`DynamicTagAnalyzer.get_tag_usage_stats()`)

---

### Story 7.3: Tag Promotion Workflow

As an **admin**,
I want **to promote a dynamic tag to the Standard Taxonomy**,
So that **frequently-used tags become official**.

**Acceptance Criteria:**

**Given** a trending dynamic tag (e.g., "topic:gdpr")
**When** admin calls `POST /v1/admin/taxonomy/promote` with:
```json
{
  "old_tag": "topic:gdpr",
  "new_tag": "compliance:regulatory:gdpr",
  "description": "GDPR compliance requirements",
  "examples": ["..."],
  "profile": "fintech",
  "add_to_taxonomy": true
}
```
**Then** the system:
1. Validates new tag format and namespace
2. Batch-updates all records: renames old_tag to new_tag in JSONB
3. Adds promotion history entry to each updated tag
4. Optionally writes new tag to taxonomy config file
5. Returns `records_updated` count and `added_to_taxonomy` status

**Technical Notes:**
- Module: `src/services/tag_governance_service.py` (`TagPromotionService`)
- Route: `src/api/routes/admin_taxonomy.py`

---

### Story 7.4: Domain Term Dictionary Management

As a **system**,
I want **a verified multi-lingual domain term dictionary**,
So that **cross-lingual retrieval is accurate**.

**Acceptance Criteria:**

**Given** domain terms extracted from documents
**When** the system processes new terms
**Then:**
1. Terms stored in `domain_term_dictionary` table with original, ja, en, vi translations
2. Translation performed by `TranslationService` using LLM (up to 3 retries)
3. Failed translations marked with `translation_failed=TRUE`
4. Terms categorized by `domain_category`

**Technical Notes:**
- Module: `src/extraction_v2/translation_service.py`
- DB: `domain_term_dictionary` table with `term_original`, `term_ja`, `term_en`, `term_vi`, `source_language`, `domain_category`, `translation_failed`

---

## Epic 8: LLM Provider Abstraction & Infrastructure

> **C-REFRESH 2026-03-21:** New epic added based on code analysis.

**Goal:** Provide provider-agnostic LLM access and infrastructure hardening.

**FRs Covered:** FR26 (Password detection), Infrastructure

---

### Story 8.1: LLM Provider Factory

As a **developer**,
I want **a provider-agnostic LLM factory**,
So that **the system can switch between Azure OpenAI and vLLM without code changes**.

**Acceptance Criteria:**

**Given** `settings.llm_provider` is set to "azure" or "vllm"
**When** any component requests an LLM client
**Then** `factory.py` returns the appropriate client:
- Azure: `AzureOpenAI`, `AzureChatOpenAI`, `AzureOpenAIEmbedding`
- vLLM: `OpenAI` (with base_url), `ChatOpenAI`, `OpenAIEmbedding`

**And** HTTP clients are fork-safe (important for Airflow workers):
- Sync: `max_connections=1`, no keepalive
- Async: `max_connections=settings.llm_concurrency`

**Technical Notes:**
- Module: `src/llm/factory.py`
- 6 factory functions: `get_chat_client`, `get_async_chat_client`, `get_chat_service`, `get_embed_model`, `get_async_embedding_client`, `get_chat_model_name`

---

### Story 8.2: Password-Protected Document Detection

As a **system**,
I want **to detect password-protected documents early in the pipeline**,
So that **users receive a clear error instead of cryptic conversion failures**.

**Acceptance Criteria:**

**Given** a user uploads a password-protected Office document
**When** the pipeline starts processing
**Then** the system:
1. Checks file magic bytes before conversion
2. Modern OOXML (.docx, .xlsx, .pptx): Detects OLE2 magic instead of ZIP
3. Legacy (.doc, .xls, .ppt): Checks for EncryptionInfo streams
4. Raises `PasswordProtectedError` with clear message
5. Sets document status to `failed` with error: "Password protected"

**Technical Notes:**
- Module: `src/extraction_v2/password_detection.py`
- Supports: `.docx`, `.docm`, `.xlsx`, `.xlsm`, `.pptx`, `.pptm`, `.doc`, `.xls`, `.ppt`

---

### Story 8.3: Domain Term Extractor Configuration

As a **developer**,
I want **configurable domain term extraction**,
So that **extraction can be tuned per deployment**.

**Acceptance Criteria:**

**Given** the domain term extractor is configured
**Then** the following are configurable via environment variables:
- `DOMAIN_TERM_MAX_PER_CHUNK` (default: 50)
- `DOMAIN_TERM_LLM_THRESHOLD` (default: 2)
- `ENABLE_CJK_SEGMENTATION` (default: true)
- `ENABLE_PERFORMANCE_MONITORING` (default: false)

**And** built-in taxonomy categories: finance, hr, engineering, legal (with extensive keyword lists)
**And** hybrid extraction: rule-based first, LLM fallback for complex content

**Technical Notes:**
- Module: `src/extraction_v2/domain_term_extractor.py`

---

## FR Coverage Matrix

| FR | Description | Epic | Story | Status |
|----|-------------|------|-------|--------|
| FR1 | Ingest Excel files | 2 | 2.2, 2.3 | Covered |
| FR2 | Batch upload | 2 | 2.4 | Covered |
| FR3 | Validate file format | 2 | 2.3 | Covered |
| FR4 | Track processing status | 2 | 2.5 | Covered |
| FR5 | Delete documents | 2 | 2.6 | Covered |
| FR6 | Detect table boundaries | 3 | 3.1 | Covered |
| FR7 | Extract multi-level headers | 3 | 3.2 | Covered |
| FR8 | Resolve merged cells | 3 | 3.3 | Covered |
| FR9 | Preserve row/column relationships | 3 | 3.1, 3.4 | Covered |
| FR10 | Handle multiple tables per sheet | 3 | 3.1 | Covered |
| FR11 | Auto-detect symbol dictionaries | 3 | 3.5 | Covered |
| FR12 | Build symbol mappings | 3 | 3.5 | Covered |
| FR13 | Resolve symbols before indexing | 3 | 3.6 | Covered |
| FR14 | Custom symbol dictionaries | 3 | 3.7 | Covered |
| FR15 | Detect cross-reference patterns | 3 | 3.8 | Covered |
| FR16 | Resolve cross-references | 3 | 3.8 | Covered |
| FR17 | Maintain cross-reference links | 3 | 3.8 | Covered |
| FR18 | Store content with metadata | 4 | 4.1, 4.2 | Covered |
| FR19 | Index for exact lookups | 4 | 4.3 | Covered |
| FR20 | Tag management (add/edit/delete) | 6 | 6.1, 6.5 | Covered |
| FR21 | HITL review workflow | 6 | 6.3 | Covered |
| FR22 | Tag history tracking | 6 | 6.1, 6.3 | Covered |
| FR23 | Trending tag analysis | 7 | 7.1, 7.2 | Covered |
| FR24 | Tag promotion workflow | 7 | 7.3 | Covered |
| FR25 | Domain term dictionary (EN/VI/JA) | 7 | 7.4 | Covered |
| FR26 | Password-protected detection | 8 | 8.2 | Covered |
| FR27 | Submit natural language questions | 5 | 5.1 | Covered |
| FR28 | Batch queries | 5 | 5.5 | Covered |
| FR29 | REST API | 1 | 1.3 | Covered |
| FR30 | API key authentication | 2 | 2.1 | Covered |
| FR31 | Rate limiting | 2 | 2.1 | Covered |
| FR32 | Request logging | 1 | 1.4 | Covered |
| FR33 | Health check endpoint | 1 | 1.5 | Covered |

**Coverage Status:** All FRs covered (100%)

---

## Summary

| Metric | Value |
|--------|-------|
| Total Epics | 8 |
| Total Stories | 45 |
| Original Epics (1-5) | 33 stories |
| New Epics (6-8, Phase 4/5) | 12 stories |
| FRs Covered | 100% |

**Epic Flow:**
```
Epic 1: Foundation → Epic 2: Upload → Epic 3: Extraction → Epic 4: Indexing → Epic 5: Q&A
        (setup)        (input)          (process)           (store)          (query)
                                            ↓
                                   Epic 6: Tagging (Phase 4) → Epic 7: Governance (Phase 5)
                                            ↓
                                   Epic 8: LLM Infra
```

---

## Change History

| Date | Version | Author | Changes |
|------|---------|--------|---------|
| 2025-11-26 | 1.0 | TextIQ | Initial epic breakdown with 5 epics, 33 stories |
| 2026-03-21 | 2.0 (Blueprint) | nguyetnta8 | C-REFRESH: Added Epic 6 (HITL Tagging - TagManagementService, TagReviewService, SemanticTagger, TagValidator), Epic 7 (Dynamic Governance - TagGovernanceService trending/stats/promote, TranslationService, domain_term_dictionary), Epic 8 (LLM Provider Factory with Azure/vLLM, password_detection.py, domain_term_extractor configuration). Updated FR Coverage Map and Matrix. Added YAML frontmatter and TOC. |

---
_Updated 2026-03-21 -- C-REFRESH sync from impl repo._