---
id: DE-PRD
title: "textiq-doc-extraction - Product Requirements Document"
version: "2.6"
status: draft
created: "2026-02-03"
lastUpdated: "2026-06-08"
author: nguyennh25
document_type: prd
service: textiq-doc-extraction
---

# textiq-doc-extraction - Product Requirements Document

**Author:** TextIQ
**Date:** 2026-02-03
**Version:** 3.1 (Hierarchical Chunking with LlamaIndex)

---

## Table of Contents

- [Executive Summary](#executive-summary)
- [Project Classification](#project-classification)
- [Success Criteria](#success-criteria)
- [Product Scope](#product-scope)
- [Functional Requirements](#functional-requirements)
- [Non-Functional Requirements](#non-functional-requirements)
- [Technical Constraints & Stack](#technical-constraints--stack)
- [LLM Provider Architecture](#llm-provider-architecture)
- [Password-Protected Document Detection](#password-protected-document-detection)
- [Tag Inheritance Engine](#tag-inheritance-engine)
- [Multi-Lingual Translation Service](#multi-lingual-translation-service)
- [Roadmap / Future Work](#roadmap--future-work)
- [Change History](#change-history)

---

## Executive Summary

textiq-doc-extraction is an LLM-powered document processing service that extracts semantic meaning from complex documents to enable accurate AI-driven Q&A. While originally focused on Excel, the system now supports a multi-format pipeline (Excel, Word, PowerPoint, PDF) using advanced conversion (Docling) and hierarchical chunking via LlamaIndex.

**The Problem:** Standard document processing tools fail on:
- Complex Excel files (flattening 2D structures)
- Non-tabular documents (losing semantic context in arbitrary chunks)
- Visual information in slides (ignoring relationships) — flowchart relationships are now preserved as inline mermaid (TEXTIQ-625)
- Large datasets (high retrieval latency and cost)
- Domain-specific terminology (poor precision in generic RAG)

**The Solution:** textiq-doc-extraction uses a V2 Extraction Pipeline with:
- **Docling Integration:** High-fidelity conversion for Large Table Records (Excel→HTML) and Word/PDF/PPTX (→Markdown).
- **ks_xlsx_parser Integration (TEXTIQ-665):** Form-layout-aware Excel→HTML→Markdown pipeline for the Details flow, replacing Docling-based Excel parsing. Inline formatting (bold, strikethrough, color spans) preserved via markdownify `_FormattedConverter`.
- **Hybrid Extraction:** LLM-based table awareness, 2-level Mermaid-Aware Chunking (LlamaIndex `MarkdownNodeParser` → branch-specific leaf splitter), and Visual Object Analysis (with flowcharts preserved as inline mermaid).
- **Domain-Aware RAG:** Automated term extraction and two-phase retrieval (Category > Term > Vector).
- **Dual-Store Architecture:** Leaf nodes in Milvus for fast similarity search, all hierarchy nodes in PostgresDocumentStore for parent-child context expansion.
- **Scalable Orchestration:** Airflow-based processing with External Python task isolation.

---

## Project Classification

**Technical Type:** api_backend
**Domain:** document_intelligence / rag_optimization
**Complexity:** medium (integrated pipelines, multi-service architecture)

---

## Success Criteria

### Primary Success Metrics

1.  **Answer Accuracy:** >=90% correct answers on Q&A test dataset.
2.  **Source Traceability:** 100% of answers include exact source reference (file, sheet/page, location).
3.  **Format Coverage:**
    - Excel: Complex tables, merged cells, multi-level headers.
    - Word/PDF: Semantic sections, tables.
    - PowerPoint: Slide content, embedded shapes, and flowcharts.
4.  **Retrieval Precision:** Reduced search space by 10-100x via Domain-Aware Filtering.

### Quality Gates

- Zero hallucinated answers (strict confidence scoring).
- Scalable processing for large tables (>1000 rows).
- Secure authentication for all backend components (Redis, API).

---

## Product Scope

### Core Capabilities (Implemented)

#### 1. Universal Document Ingestion
- **Formats:** Excel (.xlsx, .xls, .xlsm), Word (.docx, .doc, .docm), PowerPoint (.pptx), PDF.
- **Pipeline V2:** Automated format detection and routing.
- **Batch Processing:** Support for bulk uploads via Airflow.

#### 2. Advanced Extraction Strategy
**Excel (Structured — Large Table Records):**
- **Docling-based:** Converts sheets to high-fidelity HTML.
- **LLM Header Detection:** Identifies complex multi-row headers.
- **Large Table Handling:** Special efficient processing for tables >1000 rows (Chunking Strategy).
- **Symbol Resolution:** Dictionaries are detected and resolved. *(⚠️ legacy V1 — not in active DAG; superseded by V2 Docling pipeline + dual-store chunking)*

**Word/PDF/PPTX (Unstructured/Semi-struct):**
- **Docling-based:** Converts to Markdown with structural preservation.
- **2-Level Chunking (TEXTIQ-625, 2026-05-25):** LlamaIndex `MarkdownNodeParser` → leaf splitter; parent (markdown section) lives in PostgresDocumentStore for AutoMergingRetriever context expansion; 512-token leaves go to Milvus. Word/PDF/PPTX use `SentenceSplitter(chunk_size=512, paragraph_separator="\n\n")`. `.md` files share the **same mermaid-aware splitter as Excel** (`chunk_with_atomic_mermaid(target=512, max_size=EMBEDDING_MODEL_CONTEXT_TOKENS - 400)`) so flowchart fences stay inline with surrounding prose as long as the leaf fits the cap (2026-05-26 follow-up).
- **Visuals Extraction:**
    - Embeds flowcharts as inline ` ```mermaid ` blocks via per-graph serialisation of OOXML connectors (TEXTIQ-625 Goal A) — relationships and structure are preserved, not flattened to a text description.
    - Uses Azure Chat Service to generate text descriptions for non-flowchart visual elements (images, shapes).

**Excel Detailed (2-Level Mermaid-Aware, TEXTIQ-665):**
- **ks_xlsx_parser Pipeline:** Form-layout-aware XLSX parsing via `ks_xlsx_parser` → markdownify `_FormattedConverter` (HTML→MD). Preserves inline bold (`**`), strikethrough (`~~`), and color spans. One markdown section per sheet; no service-side chunking (downstream responsibility).
- **Flowchart Preservation:** Pre-parse `_detect_and_extract_flowchart` converts OOXML drawing shapes to mermaid text in cells; `collect_mermaid_blocks` / `restore_mermaid_blocks` preserves those fences through HTML→MD conversion.
- **2-Level Chunking (TEXTIQ-625):** LlamaIndex `MarkdownNodeParser` → `chunk_with_atomic_mermaid(target=512, max_size=EMBEDDING_MODEL_CONTEXT_TOKENS - 400)` so flowchart fences stay atomic when they fit the cap and oversized fences only split on body lines.
- **Dual-Store:** All nodes stored in PostgresDocumentStore; leaf nodes embedded in Milvus via LlamaIndex `VectorStoreIndex`.

#### 3. Domain-Aware Knowledge Base (RAG Optimization)
**Problem:** Naive vector search misses domain context.
**Solution:**
- **Term Extraction:** Automatically extracts domain terms (e.g., "EBITDA", "Python 3.11") from chunks via `domain_term_extractor.py`. Supports rule-based extraction with built-in taxonomy (finance, hr, engineering, legal categories) and LLM-based extraction for complex content. Configurable via `DOMAIN_TERM_MAX_PER_CHUNK` and `DOMAIN_TERM_LLM_THRESHOLD` environment variables. CJK segmentation enabled by default.
- **Taxonomy:** Categorizes chunks (e.g., "Finance", "Engineering").
- **Two-Phase Retrieval:**
    1.  **Filter:** Limit search to relevant Tenant + Category + Terms.
    2.  **Vector Search:** HNSW similarity search on reduced candidate set.

#### 4. Retrieval & Q&A
- **Hybrid Search:** Combines Text Match, Vector Similarity, and Domain Filtering.
- **Traceability:** Returns exact file, sheet, and cell range (Excel) or page/slide (Word/PDF/PPT).

#### 5. Human-in-the-Loop (HITL) Tagging (Phase 4)
- **Tag Management:** Users can Add, Edit, and Delete semantic tags on extracted records via `TagManagementService` with validation against Standard Taxonomy.
- **Review Workflow:** Staging area for "Pending" tags requiring human confirmation via `TagReviewService` (`review_tag` endpoint). Logs review history (Approver, Timestamp, Decision).
- **Auto-Verification:** Pre-defined tags (from Taxonomy) are auto-verified; new dynamic tags require review. Tag validation handled by `TagValidator` with `TaxonomyLoader`.
- **Semantic Tagger:** `semantic_tagger.py` provides LLM-based tag classification with config-driven taxonomy, supporting ISO-compliant taxonomy with Hierarchy of Intent, tag inheritance, confidence scoring, and JSONB metadata.

#### 6. Dynamic Governance (Phase 5)
- **Trending Analysis:** `TagGovernanceService` with `DynamicTagAnalyzer` identifies high-frequency dynamic tags (Shadow Taxonomy) for potential promotion. Queries tag usage statistics aggregated from JSONB tags column with min_usage, min_documents, and namespace filters.
- **Promotion Workflow:** `TagPromotionService` admin workflow to formalize dynamic tags into the controlled Standard Taxonomy. Batch-updates all records and optionally writes to taxonomy config file.
- **Domain Verification:** Multi-lingual domain term dictionary with automated translation and verification via `TranslationService`.

---

## Functional Requirements

### Document Processing
- **FR1:** System accepts Excel, Word, PowerPoint, and PDF files.
- **FR2:** System automatically detects file type and routes to appropriate V2 pipeline strategy.
- **FR3:** System converts documents for standardization: Large Table Records (Excel→HTML via Docling); Word/PDF/PPTX→Markdown via Docling; Excel Details (XLSX→HTML→Markdown via ks_xlsx_parser + markdownify `_FormattedConverter`, TEXTIQ-665).
- **FR4:** System handles large Excel tables by chunking records efficiently.
- **FR5:** System detects and extracts multi-level headers in Excel tables using LLM.
- **FR5.1:** System serialises Excel flowcharts to inline ` ```mermaid flowchart TD ` blocks (per-graph, pre-Docling) so connector relationships are preserved in chunks rather than flattened to a text description. Non-flowchart shapes (red dashed boxes, etc.) remain handled by the Azure Chat description path.

### RAG & Knowledge Base
- **FR6:** System extracts domain-specific terms and categories from all content nodes.
- **FR7:** System stores leaf nodes with embeddings in Milvus and all hierarchy nodes in PostgresDocumentStore (LlamaIndex dual-store pattern).
- **FR8:** System executes Domain-Aware Retrieval, filtering candidates before vector search.
- **FR9:** System implements 2-level chunking via LlamaIndex `MarkdownNodeParser` (parent = markdown section) → leaf splitter. The leaf splitter is `chunk_with_atomic_mermaid(target=512, max_size=EMBEDDING_MODEL_CONTEXT_TOKENS - 400)` for Excel and `.md` inputs (mermaid stays inline as atomic line-groups); Word/PDF/PPTX use `SentenceSplitter(chunk_size=512, paragraph_separator="\n\n")`. AutoMergingRetriever uses the parent chain in PostgresDocumentStore for context expansion; only leaves are embedded into Milvus. (TEXTIQ-625, collapsed from the legacy 3-level shape on 2026-05-25; .md sub-branch aligned with Excel on 2026-05-26.)
- **FR9.1:** System uses tiktoken `cl100k_base` tokenizer for token-accurate splitting compatible with text-embedding-3-large.
- **FR9.2:** *(TEXTIQ-625)* System preserves ```mermaid fences inline with surrounding content for both Excel and `.md` inputs via the shared `chunk_with_atomic_mermaid` splitter — a fence stays atomic and adjacent to its explanatory prose whenever the leaf fits `EMBEDDING_MODEL_CONTEXT_TOKENS - 400`. Fences exceeding the cap are split on body lines only (the fence boundary is never broken).
- **FR9.3:** *(TEXTIQ-627, superseded by TEXTIQ-665 for the Details flow)* Font-style ATX header injection via the `§§HN§§…§§/HN§§` sentinel scheme (openpyxl pre-Docling walk) was replaced in the Details flow by ks_xlsx_parser's own form-layout-aware structure detection. The Large Table Records flow (Docling-based) is unaffected.
- **FR9.4:** *(TEXTIQ-625)* System enforces a 3-layer embedding-budget defense: (a) chunker cap of `EMBEDDING_MODEL_CONTEXT_TOKENS - 400` (Excel) or `EMBEDDING_MODEL_CONTEXT_TOKENS` (text orphan-mermaid); (b) post-chunk `MetadataMode.EMBED` token audit logging WARNING on cap overruns; (c) `EMBED_INCLUDE_KEYS = {filename, sheet_name, document_type, element_type}` allow-list on the metadata envelope sent to the embedding API. *(TEXTIQ-665: `cell_colors`, `color_map`, and `cell_values_map` fully removed from all chunk metadata — no longer relevant to the dynamic-field envelope cap.)*

### Tagging & Governance (Phase 4 & 5)
- **FR20:** System supports renaming and deleting semantic tags on extracted chunks via the Airflow plugin API (`PATCH /api/v1/blocks/{chunk_id}/tags/{tag_id}/name`, `DELETE /api/v1/blocks/{chunk_id}/tags/{tag_id}`). Edits cascade from the LlamaIndex `data_hierarchical_nodes` JSONB store into Milvus chunk metadata.
- **FR21:** System implements HITL review workflow via tag-status transitions (`pending` -> `verified` / `rejected`) using `PATCH /api/v1/blocks/{chunk_id}/tags/status` (single) and `PATCH /api/v1/blocks/{chunk_id}/tags/bulk-status` (batch).
- **FR22:** System tracks tag history (who, when, action) for auditability.
- **FR23:** *(legacy code, deprecated — `/v1/admin/taxonomy/trending` lived on the legacy `src/api/routes/` surface; trending analysis remains a service capability via `TagGovernanceService`.)* System identifies "Trending Tags" based on frequency and document spread.
- **FR24:** *(legacy code, deprecated — `/v1/admin/taxonomy/promote` was on `src/api/routes/`; promotion remains available through `TagPromotionService` until a plugin endpoint is added.)* System promotes dynamic tags to Standard Taxonomy.
- **FR25:** System maintains a verified Domain Term Dictionary (EN/VI/JA) for cross-lingual retrieval.

### Domain Term Management (Airflow Plugin API)
- **FR27:** System supports creating domain terms individually (`POST /api/v1/domain-terms`) and in bulk (`POST /api/v1/domain-terms/bulk`) with duplicate detection scoped to tenant+node. HMAC authentication and RLS tenant isolation enforced.
- **FR28:** System supports updating domain term fields (`PATCH /api/v1/domain-terms/{term_id}`) with automatic cascade of `term_original` renames to chunk metadata in PostgreSQL and Milvus.
- **FR29:** System supports deleting domain terms (`DELETE /api/v1/domain-terms/{term_id}`) with cascade removal of the term from chunk metadata in PostgreSQL (`data_hierarchical_nodes` JSONB) and Milvus vectors. Returns 204 on success, 404 if term not found within tenant scope.

### Document Upload & Approval (Airflow Plugin API)
- **FR30:** System exposes a multipart upload endpoint (`POST /api/v1/documents/upload`) that validates extension whitelist, magic bytes, and password-protection (OOXML + OLE2) before staging the file in SeaweedFS and creating a `pending_approval` DB record. HMAC + RLS enforced.
- **FR31:** System supports approve/reject of staged uploads (`POST /api/v1/documents/{id}/approve` triggers `document_extraction_dag`; `POST /api/v1/documents/{id}/reject` removes the staged S3 object).
- **FR32:** System issues short-lived presigned S3 URLs for download (`GET /api/v1/documents/{id}/download-url`, attachment disposition) and inline PDF preview (`GET /api/v1/documents/{id}/preview-url`).
- **FR33:** System supports full document deletion (`DELETE /api/v1/documents/{id}`) with cascade across PostgreSQL, Milvus, and SeaweedFS.

### Observability
- **FR34:** System exposes a Prometheus scrape endpoint (`GET /metrics`) on the Airflow api-server with HMAC authentication. Counters cover upload/approve/reject/delete outcomes and tag-endpoint operations; histograms cover upload duration and tag-endpoint latency. Single-worker constraint (`AIRFLOW__API__WORKERS=1`) applies.

### Infrastructure & Operations
- **FR10:** System manages long-running tasks via Airflow DAGs.
- **FR11:** System supports "External Python" tasks for environment isolation.
- **FR12:** Redis is secured with authentication (Password/DB isolation).
- **FR13:** Storage implemented via SeaweedFS (S3-compatible) for scalability.

### API & Security
- **FR14:** *(legacy code, deprecated)* REST API for Upload (`/documents`) and Query (`/query`) on the legacy `src/api/` FastAPI app — superseded by the Airflow Plugin API (`/api/v1/documents/*`, see FR30-FR33). The chatbot-service / MCP search now owns query.
- **FR15:** API Key authentication with rate limiting *(legacy code, deprecated — only used by `src/api/`; the active surface uses HMAC + RLS, see FR27-FR34)*.
- **FR16:** HTTPS enforcement (NFR).

### Password-Protected Document Handling
- **FR26:** System detects password-protected Office documents (modern OOXML and legacy OLE2 formats) and fails early with a clear `PasswordProtectedError` rather than producing cryptic conversion errors.

---

## Non-Functional Requirements

### Performance
- **NFR1:** Document processing < 60s for standard files.
- **NFR2:** Large table processing scales linearly (not exponentially).
- **NFR3:** Query latency < 5s (leveraging domain filtering).

### Security
- **NFR4:** Redis protected by strong password and restricted binding.
- **NFR5:** API Keys hashed (SHA-256).
- **NFR6:** Tenant isolation via ID partitioning in database/vector store.

### Scalability
- **NFR7:** Horizontal scaling of Airflow workers (CeleryExecutor or KubernetesExecutor).
- **NFR8:** Dual-store architecture: Milvus for vector search (leaf nodes), PostgresDocumentStore for hierarchy traversal (all nodes).
- **NFR9:** SeaweedFS for distributed object storage — holds both uploaded source documents (Excel, PDF, etc.) and extracted images.

---

## Technical Constraints & Stack

- **Extraction Engine:** Docling (IBM) for Large Table Records + Word/PDF/PPTX; ks_xlsx_parser + markdownify for Excel Details (TEXTIQ-665); Custom LLM Logic.
- **Chunking & Indexing:** LlamaIndex (`MarkdownNodeParser` + branch-specific leaf splitter — `SentenceSplitter(512, "\n\n")` for text, `chunk_with_atomic_mermaid(target=512, max_size=EMBEDDING_MODEL_CONTEXT_TOKENS - 400)` for Excel; `VectorStoreIndex`, `StorageContext`, `AutoMergingRetriever`).
- **Tokenization:** tiktoken (cl100k_base encoding).
- **Backend:** Python 3.11+, FastAPI.
- **Orchestration:** Apache Airflow 3.0+.
- **Database:** PostgreSQL 16+ (structured data + LlamaIndex PostgresDocumentStore for hierarchy nodes).
- **Vector Store:** Milvus 2.4+ (LlamaIndex MilvusVectorStore, 3072-dim, COSINE).
- **Queue:** Redis 7+ (Authenticated).
- **LLM:** Azure OpenAI (GPT-4o for extraction, text-embedding-3-large for embeddings) **or** vLLM (OpenAI-compatible self-hosted).
- **Object Storage:** SeaweedFS (S3 Protocol).

---

## LLM Provider Architecture

> **C-REFRESH 2026-03-21:** Added based on code analysis of `src/llm/factory.py`.

The system implements a **provider-agnostic LLM factory pattern** that supports two backends:

| Provider | Chat Client | Embedding Client | Use Case |
|----------|-------------|-----------------|----------|
| **Azure OpenAI** | `AzureOpenAI` / `AzureChatOpenAI` (LangChain) | `AzureOpenAIEmbedding` (LlamaIndex) | Enterprise production (default) |
| **vLLM** | `OpenAI` SDK (OpenAI-compatible API) / `ChatOpenAI` (LangChain) | `OpenAIEmbedding` (LlamaIndex) | Self-hosted / development |

**Factory Functions (`src/llm/factory.py`):**
- `get_chat_client()` / `get_async_chat_client()` -- sync/async OpenAI SDK clients
- `get_chat_service()` -- LangChain `ChatService` wrapper (provider-agnostic)
- `get_embed_model()` -- LlamaIndex embedding model (Azure or vLLM)
- `get_async_embedding_client()` -- async embedding client
- `get_chat_model_name()` / `get_embedding_deployment()` -- model/deployment name resolution

**Key Design Decisions:**
- vLLM uses the standard OpenAI Python SDK with `base_url` pointed at the vLLM server (OpenAI-compatible API)
- Fork-safe HTTP clients with `httpx` (connection limits, no keepalive for sync, configurable concurrency for async via `settings.llm_concurrency`)
- Provider selection via `settings.llm_provider` and `settings.embedding_provider` configuration

---

## Password-Protected Document Detection

> **C-REFRESH 2026-03-21:** Added based on code analysis of `src/extraction_v2/password_detection.py`.

The system detects password-protected Office documents early in the pipeline to avoid cryptic downstream errors.

**Detection Strategy:**
- **Modern formats** (.docx, .xlsx, .pptx, .docm, .xlsm, .pptm): Normal files are ZIP archives (magic: `PK`). Encrypted files are OLE2 Compound Binary containers (magic: `D0 CF 11 E0 A1 B1 1A E1`).
- **Legacy formats** (.doc, .xls, .ppt): Always OLE2. Check for EncryptionInfo or encryption streams.

**Error Handling:** Raises `PasswordProtectedError` with a clear message. The pipeline catches this and sets the document status to `failed` with an appropriate error message.

---

## Tag Inheritance Engine

> **C-REFRESH 2026-03-21:** Added based on code analysis of `src/extraction_v2/inheritance_engine.py`.

The `InheritanceEngine` implements hierarchical tag cascading from document level down to section and block level (Spec Section 3.1):

- **Document-level tags:** Extracted from filename and metadata (e.g., "requirements.xlsx" -> `topic:requirements`)
- **Section-level tags:** Inferred via keyword pattern matching against section headings covering: requirements, security, performance, testing, architecture, API, database, deployment, documentation, compliance
- **Block-level tags:** Inherited from parent section/document, with source marked as `inheritance_doc` or `inheritance_section`

This enables tags to cascade without explicit manual tagging of every record.

---

## Multi-Lingual Translation Service

> **C-REFRESH 2026-03-21:** Added based on code analysis of `src/extraction_v2/translation_service.py`.

The `TranslationService` provides LLM-powered term translation for the domain term dictionary (FR25):

- **Languages:** Japanese (ja), English (en), Vietnamese (vi)
- **Backend:** Uses the LLM factory pattern (`get_chat_client()`) for provider-agnostic translation
- **Behavior:** If a term is already in a target language, it is copied as-is. If translation fails, `null` is returned for that language, and the `translation_failed` flag is set in the `domain_term_dictionary` table.
- **Retry:** Up to 3 retries on failure with structured JSON output parsing.

---

## Roadmap / Future Work

1. **GraphRAG:** Move beyond term lists to full entity relationship knowledge graphs.
2. **Multi-Modal QA:** Full Visual Question Answering (VQA) on embedded charts.
3. **Self-Correction:** Feedback loops from user ratings to update Domain Taxonomy.

---

## Change History

| Date | Version | Author | Changes |
|------|---------|--------|---------|
| 2026-02-03 | 3.0 | TextIQ | Initial PRD with hierarchical chunking and LlamaIndex |
| 2026-02-06 | 3.1 | TextIQ | Updated to reflect dual-store architecture |
| 2026-03-21 | 2.0 (Blueprint) | nguyetnta8 | C-REFRESH: Added LLM Provider Architecture (factory.py with Azure/vLLM dual-provider), Password-Protected Document Detection (password_detection.py), Tag Inheritance Engine (inheritance_engine.py), Multi-Lingual Translation Service (translation_service.py), Domain Term Extractor details, Tag Service details (TagManagementService, TagReviewService, TagGovernanceService), API route details for tags and admin_taxonomy, FR26. Added YAML frontmatter and TOC. |
| 2026-04-10 | 2.1 | nguyennh25 | Added FR27-FR29: Domain Term Management via Airflow Plugin API (create, bulk create, update with cascade, delete with cascade). HMAC auth + RLS tenant isolation. |
| 2026-05-06 | 2.2 | nguyennh25 | Re-aligned FR20-FR21 with the implemented chunk-tag endpoints (`PATCH /api/v1/blocks/{chunk_id}/tags/status`, `/bulk-status`, `/{tag_id}/name`, `DELETE /{tag_id}`); added FR30-FR33 (Document Upload & Approval — `/upload`, `/{id}/approve|reject`, presigned download/preview URLs, `DELETE /{id}`) and FR34 (Prometheus `/metrics` endpoint, HMAC-protected). |
| 2026-05-06 | 2.3 | nguyennh25 | Three-part doc pass. **(a) src/api routes:** marked the legacy `src/api/{main.py, dependencies.py, middleware.py, routes/*}` surface as legacy code, deprecated — FR14 (`/documents`, `/query`) and FR23/FR24 (`/v1/admin/taxonomy/*`) now carry a "legacy code, deprecated" note pointing at the Airflow Plugin API equivalents (FR30-FR34). `src/api/schemas/*` remains live (shared data classes used by tagging pipeline). **(b) Legacy V1 capability:** marked the Excel "Symbol Resolution" extraction-strategy bullet as legacy V1 — not in active DAG; superseded by the V2 Docling pipeline + dual-store chunking. Aligns with the architecture-side dead-code pass that flagged `src/extraction/`, `src/adapters/legacy_to_milvus_adapter.py`, the orphaned `processing_tasks.py` tasks, and the `extracted_records` / `symbol_dictionaries` / `cross_references` PG tables as legacy V1 only. **(c) SeaweedFS clarification:** updated NFR9 to make explicit that SeaweedFS holds **both** uploaded source documents (Excel, PDF, etc.) and extracted images. |
| 2026-06-08 | 2.6 | nguyennh25 | TEXTIQ-665: Excel Details flow now uses ks_xlsx_parser + markdownify `_FormattedConverter` instead of Docling. Dropped: Docling DocumentConverter, CellColorExtractor, sentinel injection (TEXTIQ-627 superseded for Details flow), `color_map`/`cell_values_map`/`cell_colors` from all chunk metadata. Kept: `_detect_and_extract_flowchart` (shapes→mermaid) + `collect_mermaid_blocks`/`restore_mermaid_blocks`. LibreOffice conversion timeout bumped to 300s. `parse_excel_file` returns one DTO per sheet (no service-side chunking). Updated FR3, FR9.3, FR9.4, Technical Constraints. Promoted `src/extraction_v2/ks_parser/` package. |
| 2026-05-26 | 2.5 | nguyennh25 | TEXTIQ-625 md-branch follow-up: `.md` inputs now use the same mermaid-aware splitter as Excel (`chunk_with_atomic_mermaid(target=512, max_size=EMBEDDING_MODEL_CONTEXT_TOKENS - 400)`) so flowchart fences stay inline with surrounding prose, not orphaned as standalone leaves. Updated the Word/PDF/PPTX + Excel Detailed "Advanced Extraction Strategy" bullet (the `.md` sub-branch now matches Excel), FR9 (leaf-splitter description rewritten to "Excel and `.md` use `chunk_with_atomic_mermaid`; Word/PDF/PPTX use `SentenceSplitter`"), and FR9.2 (preservation guarantee now covers both Excel and `.md`). |
| 2026-05-25 | 2.4 | nguyennh25 | Synced PRD with three implemented specs. **(a) TEXTIQ-625 (mermaid pipeline):** Excel flowcharts are now serialised to inline ` ```mermaid ` fences during pre-Docling OOXML processing — updated the Executive Summary problem statement (L46), the Word/PDF/PPTX and Excel Detailed "Advanced Extraction Strategy" bullets (L105/107/111), and FR5.1 (L148) to reflect mermaid embedding rather than NL flowchart description. **(b) TEXTIQ-625 (unified 2-level chunking):** FR9 (L154) rewritten from "3-level node hierarchy" to "2-level MarkdownNodeParser → branch-specific leaf splitter (Excel: `chunk_with_atomic_mermaid` target=512 / max≈7791; text: `SentenceSplitter(512, "\\n\\n")`)". Added FR9.2 (in-context mermaid preservation), FR9.3 (TEXTIQ-627 font-style ATX header injection), FR9.4 (4-layer embedding-budget defense: chunker cap, post-chunk audit, `EMBED_INCLUDE_KEYS` allow-list, parent-only `cell_colors`). **(c) TEXTIQ-627 (font-style ATX header injection):** validated by smoke run on the IF workbook (184 chunks, 90.2% non-trivial header_path, max 5274 tokens, 22 mermaid chunks preserved, 0 chunks exceed 8191 tokens). |

---
_Updated 2026-06-08 -- TEXTIQ-665: Excel Details flow replaced with ks_xlsx_parser + markdownify; cell_colors/color_map dropped; sentinel injection superseded._
