---
id: DE-BRIEF
title: "textiq-doc-extraction - Product Brief"
version: "2.0"
status: draft
created: "2025-11-26"
lastUpdated: "2026-03-21"
author: nguyetnta8
document_type: product-brief
service: textiq-doc-extraction
---

# textiq-doc-extraction - Product Brief

## Table of Contents

- [Service Identity](#service-identity)
- [Problem Statement](#problem-statement)
- [Solution Overview](#solution-overview)
- [Core Capabilities](#core-capabilities)
- [Technical Stack](#technical-stack)
- [Architecture Highlights](#architecture-highlights)
- [API Surface](#api-surface)
- [Key Differentiators](#key-differentiators)
- [Current Status](#current-status)
- [Change History](#change-history)

---

## Service Identity

| Field | Value |
|-------|-------|
| **Service Name** | textiq-doc-extraction |
| **Domain** | Document Intelligence / RAG Optimization |
| **Type** | API Backend (Python microservice) |
| **Owner** | NguyetNTA8 |
| **Repository** | textiq-doc-extraction |
| **Stack** | Python 3.11+, FastAPI, Airflow 3.x, PostgreSQL 18, Milvus 2.4 |

---

## Problem Statement

Enterprise document processing tools fail on:
1. **Complex Excel tables** -- flattened 2D structures lose semantic relationships
2. **Non-tabular documents** -- arbitrary chunking loses section context
3. **Visual content** -- flowcharts, shapes, and embedded images are ignored
4. **Domain terminology** -- generic RAG produces poor precision on specialized vocabulary
5. **Scale** -- naive vector search over large datasets has high latency and cost

---

## Solution Overview

textiq-doc-extraction is an LLM-powered document processing pipeline that extracts semantic meaning from complex documents (Excel, Word, PDF, PowerPoint) to enable accurate AI-driven Q&A.

**Key innovations:**
- **Dual Excel pipeline** -- parallel Large Table (Docling) + Detailed (Docling+openpyxl) extraction
- **Hierarchical chunking** -- LlamaIndex 3-level node hierarchy with dual-store pattern
- **Domain-Aware RAG** -- automated term extraction + two-phase retrieval (filter then search)
- **HITL tagging** -- human-in-the-loop tag management with review workflow
- **Dynamic governance** -- trending tag analysis and promotion to Standard Taxonomy
- **Provider-agnostic LLM** -- factory pattern supporting Azure OpenAI and vLLM

---

## Core Capabilities

### 1. Universal Document Ingestion
- Formats: Excel (.xlsx/.xls/.xlsm), Word (.docx/.doc/.docm), PowerPoint (.pptx), PDF
- Automated format detection and routing
- Password-protected document detection (early fail with clear error)
- Batch processing via Airflow DAGs

### 2. Advanced Extraction
- **Excel Large Table:** Docling HTML conversion, LLM header detection, row-level records
- **Excel Detailed:** Hybrid Docling+openpyxl for images, shapes, flowcharts with AI descriptions
- **Text Documents:** Docling Markdown conversion with hierarchical chunking [2048/512/256]
- **Symbol Resolution:** Auto-detected dictionaries, cross-reference linking

### 3. Domain-Aware Knowledge Base
- Automated domain term extraction (rule-based + LLM hybrid, CJK support)
- Multi-lingual domain term dictionary (EN/VI/JA) with LLM-powered translation
- Two-phase retrieval: category/term pre-filter then HNSW vector search
- Dual-store: Milvus (leaf embeddings) + PostgresDocumentStore (full hierarchy)

### 4. Semantic Tagging (Phase 4)
- LLM-based automatic tag classification via SemanticTagger
- Tag inheritance engine: document -> section -> block cascading
- HITL review workflow: pending -> confirmed/rejected with audit trail
- Tag validation against Standard Taxonomy with profile support

### 5. Dynamic Governance (Phase 5)
- Trending tag analysis (usage frequency, document spread, verified ratio)
- Tag promotion workflow: dynamic -> canonical with batch record updates
- Admin dashboard API: trending, stats, promote endpoints
- Taxonomy config file management

### 6. Q&A & Retrieval
- Hybrid search: text match + vector similarity + domain filtering
- Confidence scoring with multi-factor calculation
- Source traceability: exact file/sheet/row or page/slide references
- Batch query support (up to 10 concurrent)

---

## Technical Stack

| Component | Technology | Purpose |
|-----------|-----------|---------|
| Backend | Python 3.11+, FastAPI | API + business logic |
| Orchestration | Apache Airflow 3.x | Pipeline workflow management |
| Database | PostgreSQL 18 | Structured data + LlamaIndex DocStore |
| Vector Store | Milvus 2.4 | Leaf node embeddings (3072-dim, COSINE) |
| LLM | Azure OpenAI GPT-4o / vLLM | Extraction, tagging, Q&A, translation |
| Embeddings | text-embedding-3-large (3072d) | Vector generation |
| Object Storage | SeaweedFS | Extracted images |
| Queue | Redis 7+ | Airflow backend |
| Extraction | Docling + openpyxl | Document conversion |
| Chunking | LlamaIndex HierarchicalNodeParser | Multi-level document hierarchy |
| Packages | uv | Fast Python package management |

---

## Architecture Highlights

```
Client -> FastAPI -> Airflow DAG -> Extraction Pipeline
                                      |
                                      ├── Excel Large Table (Docling) -> Milvus
                                      ├── Excel Detailed (Docling+openpyxl) -> Milvus + DocStore
                                      └── Text (Docling) -> Milvus + DocStore
                                      |
                            Domain Terms + Semantic Tags enrichment at each stage
```

**LLM Factory Pattern:** Provider-agnostic access via `src/llm/factory.py` supporting Azure OpenAI and vLLM with fork-safe HTTP clients.

**DB Schema Highlights:**
- `extracted_records.tags` -- JSONB array (not SQL ARRAY) for tag storage with history
- `domain_term_dictionary` -- multi-lingual term store with translation status
- `hierarchical_nodes` -- LlamaIndex managed table for node hierarchy

---

## API Surface

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/documents` | POST | Upload document |
| `/documents/batch` | POST | Batch upload |
| `/documents/{id}` | GET/DELETE | Status/delete |
| `/query` | POST | Q&A query |
| `/query/batch` | POST | Batch queries |
| `/health` | GET | Health check |
| `/v1/blocks/{id}/tags` | GET/POST | Tag management |
| `/v1/blocks/{id}/tags/review` | POST | HITL tag review |
| `/v1/admin/taxonomy/trending` | GET | Trending tags |
| `/v1/admin/taxonomy/stats` | GET | Tag statistics |
| `/v1/admin/taxonomy/promote` | POST | Tag promotion |

---

## Key Differentiators

1. **Dual Excel pipeline** preserves both table structure AND visual content
2. **Hierarchical chunking** enables context expansion at query time via AutoMergingRetriever
3. **Domain-aware retrieval** reduces search space 10-100x before vector search
4. **HITL tagging with governance** ensures taxonomy evolves with usage
5. **Provider-agnostic LLM** supports enterprise Azure and self-hosted vLLM
6. **Password-protected document detection** fails early with clear errors

---

## Current Status

- **Epics 1-5:** Core pipeline implemented (foundation, upload, extraction, indexing, Q&A)
- **Epic 6 (Phase 4):** HITL tagging implemented (TagManagement, TagReview, SemanticTagger)
- **Epic 7 (Phase 5):** Dynamic governance implemented (trending, stats, promote, translation)
- **Epic 8:** LLM factory (Azure/vLLM), password detection, domain term config implemented

---

## Change History

| Date | Version | Author | Changes |
|------|---------|--------|---------|
| 2025-11-26 | 1.0 | TextIQ | Initial product brief |
| 2026-03-21 | 2.0 (Blueprint) | nguyetnta8 | C-REFRESH: Complete rewrite incorporating Phase 4/5 capabilities (tag services, governance, translation), LLM provider factory (Azure/vLLM), password detection, domain term extractor details, API surface update, architecture highlights. Added YAML frontmatter and TOC. |

---
_Updated 2026-03-21 -- C-REFRESH sync from impl repo._