--- id: DE-PRD title: "textiq-doc-extraction - Product Requirements Document" version: "2.6" status: draft created: "2026-02-03" lastUpdated: "2026-06-08" author: nguyennh25 document_type: prd service: textiq-doc-extraction --- # textiq-doc-extraction - Product Requirements Document **Author:** TextIQ **Date:** 2026-02-03 **Version:** 3.1 (Hierarchical Chunking with LlamaIndex) --- ## Table of Contents - [Executive Summary](#executive-summary) - [Project Classification](#project-classification) - [Success Criteria](#success-criteria) - [Product Scope](#product-scope) - [Functional Requirements](#functional-requirements) - [Non-Functional Requirements](#non-functional-requirements) - [Technical Constraints & Stack](#technical-constraints--stack) - [LLM Provider Architecture](#llm-provider-architecture) - [Password-Protected Document Detection](#password-protected-document-detection) - [Tag Inheritance Engine](#tag-inheritance-engine) - [Multi-Lingual Translation Service](#multi-lingual-translation-service) - [Roadmap / Future Work](#roadmap--future-work) - [Change History](#change-history) --- ## Executive Summary textiq-doc-extraction is an LLM-powered document processing service that extracts semantic meaning from complex documents to enable accurate AI-driven Q&A. While originally focused on Excel, the system now supports a multi-format pipeline (Excel, Word, PowerPoint, PDF) using advanced conversion (Docling) and hierarchical chunking via LlamaIndex. **The Problem:** Standard document processing tools fail on: - Complex Excel files (flattening 2D structures) - Non-tabular documents (losing semantic context in arbitrary chunks) - Visual information in slides (ignoring relationships) — flowchart relationships are now preserved as inline mermaid (TEXTIQ-625) - Large datasets (high retrieval latency and cost) - Domain-specific terminology (poor precision in generic RAG) **The Solution:** textiq-doc-extraction uses a V2 Extraction Pipeline with: - **Docling Integration:** High-fidelity conversion for Large Table Records (Excel→HTML) and Word/PDF/PPTX (→Markdown). - **ks_xlsx_parser Integration (TEXTIQ-665):** Form-layout-aware Excel→HTML→Markdown pipeline for the Details flow, replacing Docling-based Excel parsing. Inline formatting (bold, strikethrough, color spans) preserved via markdownify `_FormattedConverter`. - **Hybrid Extraction:** LLM-based table awareness, 2-level Mermaid-Aware Chunking (LlamaIndex `MarkdownNodeParser` → branch-specific leaf splitter), and Visual Object Analysis (with flowcharts preserved as inline mermaid). - **Domain-Aware RAG:** Automated term extraction and two-phase retrieval (Category > Term > Vector). - **Dual-Store Architecture:** Leaf nodes in Milvus for fast similarity search, all hierarchy nodes in PostgresDocumentStore for parent-child context expansion. - **Scalable Orchestration:** Airflow-based processing with External Python task isolation. --- ## Project Classification **Technical Type:** api_backend **Domain:** document_intelligence / rag_optimization **Complexity:** medium (integrated pipelines, multi-service architecture) --- ## Success Criteria ### Primary Success Metrics 1. **Answer Accuracy:** >=90% correct answers on Q&A test dataset. 2. **Source Traceability:** 100% of answers include exact source reference (file, sheet/page, location). 3. **Format Coverage:** - Excel: Complex tables, merged cells, multi-level headers. - Word/PDF: Semantic sections, tables. - PowerPoint: Slide content, embedded shapes, and flowcharts. 4. **Retrieval Precision:** Reduced search space by 10-100x via Domain-Aware Filtering. ### Quality Gates - Zero hallucinated answers (strict confidence scoring). - Scalable processing for large tables (>1000 rows). - Secure authentication for all backend components (Redis, API). --- ## Product Scope ### Core Capabilities (Implemented) #### 1. Universal Document Ingestion - **Formats:** Excel (.xlsx, .xls, .xlsm), Word (.docx, .doc, .docm), PowerPoint (.pptx), PDF. - **Pipeline V2:** Automated format detection and routing. - **Batch Processing:** Support for bulk uploads via Airflow. #### 2. Advanced Extraction Strategy **Excel (Structured — Large Table Records):** - **Docling-based:** Converts sheets to high-fidelity HTML. - **LLM Header Detection:** Identifies complex multi-row headers. - **Large Table Handling:** Special efficient processing for tables >1000 rows (Chunking Strategy). - **Symbol Resolution:** Dictionaries are detected and resolved. *(⚠️ legacy V1 — not in active DAG; superseded by V2 Docling pipeline + dual-store chunking)* **Word/PDF/PPTX (Unstructured/Semi-struct):** - **Docling-based:** Converts to Markdown with structural preservation. - **2-Level Chunking (TEXTIQ-625, 2026-05-25):** LlamaIndex `MarkdownNodeParser` → leaf splitter; parent (markdown section) lives in PostgresDocumentStore for AutoMergingRetriever context expansion; 512-token leaves go to Milvus. Word/PDF/PPTX use `SentenceSplitter(chunk_size=512, paragraph_separator="\n\n")`. `.md` files share the **same mermaid-aware splitter as Excel** (`chunk_with_atomic_mermaid(target=512, max_size=EMBEDDING_MODEL_CONTEXT_TOKENS - 400)`) so flowchart fences stay inline with surrounding prose as long as the leaf fits the cap (2026-05-26 follow-up). - **Visuals Extraction:** - Embeds flowcharts as inline ` ```mermaid ` blocks via per-graph serialisation of OOXML connectors (TEXTIQ-625 Goal A) — relationships and structure are preserved, not flattened to a text description. - Uses Azure Chat Service to generate text descriptions for non-flowchart visual elements (images, shapes). **Excel Detailed (2-Level Mermaid-Aware, TEXTIQ-665):** - **ks_xlsx_parser Pipeline:** Form-layout-aware XLSX parsing via `ks_xlsx_parser` → markdownify `_FormattedConverter` (HTML→MD). Preserves inline bold (`**`), strikethrough (`~~`), and color spans. One markdown section per sheet; no service-side chunking (downstream responsibility). - **Flowchart Preservation:** Pre-parse `_detect_and_extract_flowchart` converts OOXML drawing shapes to mermaid text in cells; `collect_mermaid_blocks` / `restore_mermaid_blocks` preserves those fences through HTML→MD conversion. - **2-Level Chunking (TEXTIQ-625):** LlamaIndex `MarkdownNodeParser` → `chunk_with_atomic_mermaid(target=512, max_size=EMBEDDING_MODEL_CONTEXT_TOKENS - 400)` so flowchart fences stay atomic when they fit the cap and oversized fences only split on body lines. - **Dual-Store:** All nodes stored in PostgresDocumentStore; leaf nodes embedded in Milvus via LlamaIndex `VectorStoreIndex`. #### 3. Domain-Aware Knowledge Base (RAG Optimization) **Problem:** Naive vector search misses domain context. **Solution:** - **Term Extraction:** Automatically extracts domain terms (e.g., "EBITDA", "Python 3.11") from chunks via `domain_term_extractor.py`. Supports rule-based extraction with built-in taxonomy (finance, hr, engineering, legal categories) and LLM-based extraction for complex content. Configurable via `DOMAIN_TERM_MAX_PER_CHUNK` and `DOMAIN_TERM_LLM_THRESHOLD` environment variables. CJK segmentation enabled by default. - **Taxonomy:** Categorizes chunks (e.g., "Finance", "Engineering"). - **Two-Phase Retrieval:** 1. **Filter:** Limit search to relevant Tenant + Category + Terms. 2. **Vector Search:** HNSW similarity search on reduced candidate set. #### 4. Retrieval & Q&A - **Hybrid Search:** Combines Text Match, Vector Similarity, and Domain Filtering. - **Traceability:** Returns exact file, sheet, and cell range (Excel) or page/slide (Word/PDF/PPT). #### 5. Human-in-the-Loop (HITL) Tagging (Phase 4) - **Tag Management:** Users can Add, Edit, and Delete semantic tags on extracted records via `TagManagementService` with validation against Standard Taxonomy. - **Review Workflow:** Staging area for "Pending" tags requiring human confirmation via `TagReviewService` (`review_tag` endpoint). Logs review history (Approver, Timestamp, Decision). - **Auto-Verification:** Pre-defined tags (from Taxonomy) are auto-verified; new dynamic tags require review. Tag validation handled by `TagValidator` with `TaxonomyLoader`. - **Semantic Tagger:** `semantic_tagger.py` provides LLM-based tag classification with config-driven taxonomy, supporting ISO-compliant taxonomy with Hierarchy of Intent, tag inheritance, confidence scoring, and JSONB metadata. #### 6. Dynamic Governance (Phase 5) - **Trending Analysis:** `TagGovernanceService` with `DynamicTagAnalyzer` identifies high-frequency dynamic tags (Shadow Taxonomy) for potential promotion. Queries tag usage statistics aggregated from JSONB tags column with min_usage, min_documents, and namespace filters. - **Promotion Workflow:** `TagPromotionService` admin workflow to formalize dynamic tags into the controlled Standard Taxonomy. Batch-updates all records and optionally writes to taxonomy config file. - **Domain Verification:** Multi-lingual domain term dictionary with automated translation and verification via `TranslationService`. --- ## Functional Requirements ### Document Processing - **FR1:** System accepts Excel, Word, PowerPoint, and PDF files. - **FR2:** System automatically detects file type and routes to appropriate V2 pipeline strategy. - **FR3:** System converts documents for standardization: Large Table Records (Excel→HTML via Docling); Word/PDF/PPTX→Markdown via Docling; Excel Details (XLSX→HTML→Markdown via ks_xlsx_parser + markdownify `_FormattedConverter`, TEXTIQ-665). - **FR4:** System handles large Excel tables by chunking records efficiently. - **FR5:** System detects and extracts multi-level headers in Excel tables using LLM. - **FR5.1:** System serialises Excel flowcharts to inline ` ```mermaid flowchart TD ` blocks (per-graph, pre-Docling) so connector relationships are preserved in chunks rather than flattened to a text description. Non-flowchart shapes (red dashed boxes, etc.) remain handled by the Azure Chat description path. ### RAG & Knowledge Base - **FR6:** System extracts domain-specific terms and categories from all content nodes. - **FR7:** System stores leaf nodes with embeddings in Milvus and all hierarchy nodes in PostgresDocumentStore (LlamaIndex dual-store pattern). - **FR8:** System executes Domain-Aware Retrieval, filtering candidates before vector search. - **FR9:** System implements 2-level chunking via LlamaIndex `MarkdownNodeParser` (parent = markdown section) → leaf splitter. The leaf splitter is `chunk_with_atomic_mermaid(target=512, max_size=EMBEDDING_MODEL_CONTEXT_TOKENS - 400)` for Excel and `.md` inputs (mermaid stays inline as atomic line-groups); Word/PDF/PPTX use `SentenceSplitter(chunk_size=512, paragraph_separator="\n\n")`. AutoMergingRetriever uses the parent chain in PostgresDocumentStore for context expansion; only leaves are embedded into Milvus. (TEXTIQ-625, collapsed from the legacy 3-level shape on 2026-05-25; .md sub-branch aligned with Excel on 2026-05-26.) - **FR9.1:** System uses tiktoken `cl100k_base` tokenizer for token-accurate splitting compatible with text-embedding-3-large. - **FR9.2:** *(TEXTIQ-625)* System preserves ```mermaid fences inline with surrounding content for both Excel and `.md` inputs via the shared `chunk_with_atomic_mermaid` splitter — a fence stays atomic and adjacent to its explanatory prose whenever the leaf fits `EMBEDDING_MODEL_CONTEXT_TOKENS - 400`. Fences exceeding the cap are split on body lines only (the fence boundary is never broken). - **FR9.3:** *(TEXTIQ-627, superseded by TEXTIQ-665 for the Details flow)* Font-style ATX header injection via the `§§HN§§…§§/HN§§` sentinel scheme (openpyxl pre-Docling walk) was replaced in the Details flow by ks_xlsx_parser's own form-layout-aware structure detection. The Large Table Records flow (Docling-based) is unaffected. - **FR9.4:** *(TEXTIQ-625)* System enforces a 3-layer embedding-budget defense: (a) chunker cap of `EMBEDDING_MODEL_CONTEXT_TOKENS - 400` (Excel) or `EMBEDDING_MODEL_CONTEXT_TOKENS` (text orphan-mermaid); (b) post-chunk `MetadataMode.EMBED` token audit logging WARNING on cap overruns; (c) `EMBED_INCLUDE_KEYS = {filename, sheet_name, document_type, element_type}` allow-list on the metadata envelope sent to the embedding API. *(TEXTIQ-665: `cell_colors`, `color_map`, and `cell_values_map` fully removed from all chunk metadata — no longer relevant to the dynamic-field envelope cap.)* ### Tagging & Governance (Phase 4 & 5) - **FR20:** System supports renaming and deleting semantic tags on extracted chunks via the Airflow plugin API (`PATCH /api/v1/blocks/{chunk_id}/tags/{tag_id}/name`, `DELETE /api/v1/blocks/{chunk_id}/tags/{tag_id}`). Edits cascade from the LlamaIndex `data_hierarchical_nodes` JSONB store into Milvus chunk metadata. - **FR21:** System implements HITL review workflow via tag-status transitions (`pending` -> `verified` / `rejected`) using `PATCH /api/v1/blocks/{chunk_id}/tags/status` (single) and `PATCH /api/v1/blocks/{chunk_id}/tags/bulk-status` (batch). - **FR22:** System tracks tag history (who, when, action) for auditability. - **FR23:** *(legacy code, deprecated — `/v1/admin/taxonomy/trending` lived on the legacy `src/api/routes/` surface; trending analysis remains a service capability via `TagGovernanceService`.)* System identifies "Trending Tags" based on frequency and document spread. - **FR24:** *(legacy code, deprecated — `/v1/admin/taxonomy/promote` was on `src/api/routes/`; promotion remains available through `TagPromotionService` until a plugin endpoint is added.)* System promotes dynamic tags to Standard Taxonomy. - **FR25:** System maintains a verified Domain Term Dictionary (EN/VI/JA) for cross-lingual retrieval. ### Domain Term Management (Airflow Plugin API) - **FR27:** System supports creating domain terms individually (`POST /api/v1/domain-terms`) and in bulk (`POST /api/v1/domain-terms/bulk`) with duplicate detection scoped to tenant+node. HMAC authentication and RLS tenant isolation enforced. - **FR28:** System supports updating domain term fields (`PATCH /api/v1/domain-terms/{term_id}`) with automatic cascade of `term_original` renames to chunk metadata in PostgreSQL and Milvus. - **FR29:** System supports deleting domain terms (`DELETE /api/v1/domain-terms/{term_id}`) with cascade removal of the term from chunk metadata in PostgreSQL (`data_hierarchical_nodes` JSONB) and Milvus vectors. Returns 204 on success, 404 if term not found within tenant scope. ### Document Upload & Approval (Airflow Plugin API) - **FR30:** System exposes a multipart upload endpoint (`POST /api/v1/documents/upload`) that validates extension whitelist, magic bytes, and password-protection (OOXML + OLE2) before staging the file in SeaweedFS and creating a `pending_approval` DB record. HMAC + RLS enforced. - **FR31:** System supports approve/reject of staged uploads (`POST /api/v1/documents/{id}/approve` triggers `document_extraction_dag`; `POST /api/v1/documents/{id}/reject` removes the staged S3 object). - **FR32:** System issues short-lived presigned S3 URLs for download (`GET /api/v1/documents/{id}/download-url`, attachment disposition) and inline PDF preview (`GET /api/v1/documents/{id}/preview-url`). - **FR33:** System supports full document deletion (`DELETE /api/v1/documents/{id}`) with cascade across PostgreSQL, Milvus, and SeaweedFS. ### Observability - **FR34:** System exposes a Prometheus scrape endpoint (`GET /metrics`) on the Airflow api-server with HMAC authentication. Counters cover upload/approve/reject/delete outcomes and tag-endpoint operations; histograms cover upload duration and tag-endpoint latency. Single-worker constraint (`AIRFLOW__API__WORKERS=1`) applies. ### Infrastructure & Operations - **FR10:** System manages long-running tasks via Airflow DAGs. - **FR11:** System supports "External Python" tasks for environment isolation. - **FR12:** Redis is secured with authentication (Password/DB isolation). - **FR13:** Storage implemented via SeaweedFS (S3-compatible) for scalability. ### API & Security - **FR14:** *(legacy code, deprecated)* REST API for Upload (`/documents`) and Query (`/query`) on the legacy `src/api/` FastAPI app — superseded by the Airflow Plugin API (`/api/v1/documents/*`, see FR30-FR33). The chatbot-service / MCP search now owns query. - **FR15:** API Key authentication with rate limiting *(legacy code, deprecated — only used by `src/api/`; the active surface uses HMAC + RLS, see FR27-FR34)*. - **FR16:** HTTPS enforcement (NFR). ### Password-Protected Document Handling - **FR26:** System detects password-protected Office documents (modern OOXML and legacy OLE2 formats) and fails early with a clear `PasswordProtectedError` rather than producing cryptic conversion errors. --- ## Non-Functional Requirements ### Performance - **NFR1:** Document processing < 60s for standard files. - **NFR2:** Large table processing scales linearly (not exponentially). - **NFR3:** Query latency < 5s (leveraging domain filtering). ### Security - **NFR4:** Redis protected by strong password and restricted binding. - **NFR5:** API Keys hashed (SHA-256). - **NFR6:** Tenant isolation via ID partitioning in database/vector store. ### Scalability - **NFR7:** Horizontal scaling of Airflow workers (CeleryExecutor or KubernetesExecutor). - **NFR8:** Dual-store architecture: Milvus for vector search (leaf nodes), PostgresDocumentStore for hierarchy traversal (all nodes). - **NFR9:** SeaweedFS for distributed object storage — holds both uploaded source documents (Excel, PDF, etc.) and extracted images. --- ## Technical Constraints & Stack - **Extraction Engine:** Docling (IBM) for Large Table Records + Word/PDF/PPTX; ks_xlsx_parser + markdownify for Excel Details (TEXTIQ-665); Custom LLM Logic. - **Chunking & Indexing:** LlamaIndex (`MarkdownNodeParser` + branch-specific leaf splitter — `SentenceSplitter(512, "\n\n")` for text, `chunk_with_atomic_mermaid(target=512, max_size=EMBEDDING_MODEL_CONTEXT_TOKENS - 400)` for Excel; `VectorStoreIndex`, `StorageContext`, `AutoMergingRetriever`). - **Tokenization:** tiktoken (cl100k_base encoding). - **Backend:** Python 3.11+, FastAPI. - **Orchestration:** Apache Airflow 3.0+. - **Database:** PostgreSQL 16+ (structured data + LlamaIndex PostgresDocumentStore for hierarchy nodes). - **Vector Store:** Milvus 2.4+ (LlamaIndex MilvusVectorStore, 3072-dim, COSINE). - **Queue:** Redis 7+ (Authenticated). - **LLM:** Azure OpenAI (GPT-4o for extraction, text-embedding-3-large for embeddings) **or** vLLM (OpenAI-compatible self-hosted). - **Object Storage:** SeaweedFS (S3 Protocol). --- ## LLM Provider Architecture > **C-REFRESH 2026-03-21:** Added based on code analysis of `src/llm/factory.py`. The system implements a **provider-agnostic LLM factory pattern** that supports two backends: | Provider | Chat Client | Embedding Client | Use Case | |----------|-------------|-----------------|----------| | **Azure OpenAI** | `AzureOpenAI` / `AzureChatOpenAI` (LangChain) | `AzureOpenAIEmbedding` (LlamaIndex) | Enterprise production (default) | | **vLLM** | `OpenAI` SDK (OpenAI-compatible API) / `ChatOpenAI` (LangChain) | `OpenAIEmbedding` (LlamaIndex) | Self-hosted / development | **Factory Functions (`src/llm/factory.py`):** - `get_chat_client()` / `get_async_chat_client()` -- sync/async OpenAI SDK clients - `get_chat_service()` -- LangChain `ChatService` wrapper (provider-agnostic) - `get_embed_model()` -- LlamaIndex embedding model (Azure or vLLM) - `get_async_embedding_client()` -- async embedding client - `get_chat_model_name()` / `get_embedding_deployment()` -- model/deployment name resolution **Key Design Decisions:** - vLLM uses the standard OpenAI Python SDK with `base_url` pointed at the vLLM server (OpenAI-compatible API) - Fork-safe HTTP clients with `httpx` (connection limits, no keepalive for sync, configurable concurrency for async via `settings.llm_concurrency`) - Provider selection via `settings.llm_provider` and `settings.embedding_provider` configuration --- ## Password-Protected Document Detection > **C-REFRESH 2026-03-21:** Added based on code analysis of `src/extraction_v2/password_detection.py`. The system detects password-protected Office documents early in the pipeline to avoid cryptic downstream errors. **Detection Strategy:** - **Modern formats** (.docx, .xlsx, .pptx, .docm, .xlsm, .pptm): Normal files are ZIP archives (magic: `PK`). Encrypted files are OLE2 Compound Binary containers (magic: `D0 CF 11 E0 A1 B1 1A E1`). - **Legacy formats** (.doc, .xls, .ppt): Always OLE2. Check for EncryptionInfo or encryption streams. **Error Handling:** Raises `PasswordProtectedError` with a clear message. The pipeline catches this and sets the document status to `failed` with an appropriate error message. --- ## Tag Inheritance Engine > **C-REFRESH 2026-03-21:** Added based on code analysis of `src/extraction_v2/inheritance_engine.py`. The `InheritanceEngine` implements hierarchical tag cascading from document level down to section and block level (Spec Section 3.1): - **Document-level tags:** Extracted from filename and metadata (e.g., "requirements.xlsx" -> `topic:requirements`) - **Section-level tags:** Inferred via keyword pattern matching against section headings covering: requirements, security, performance, testing, architecture, API, database, deployment, documentation, compliance - **Block-level tags:** Inherited from parent section/document, with source marked as `inheritance_doc` or `inheritance_section` This enables tags to cascade without explicit manual tagging of every record. --- ## Multi-Lingual Translation Service > **C-REFRESH 2026-03-21:** Added based on code analysis of `src/extraction_v2/translation_service.py`. The `TranslationService` provides LLM-powered term translation for the domain term dictionary (FR25): - **Languages:** Japanese (ja), English (en), Vietnamese (vi) - **Backend:** Uses the LLM factory pattern (`get_chat_client()`) for provider-agnostic translation - **Behavior:** If a term is already in a target language, it is copied as-is. If translation fails, `null` is returned for that language, and the `translation_failed` flag is set in the `domain_term_dictionary` table. - **Retry:** Up to 3 retries on failure with structured JSON output parsing. --- ## Roadmap / Future Work 1. **GraphRAG:** Move beyond term lists to full entity relationship knowledge graphs. 2. **Multi-Modal QA:** Full Visual Question Answering (VQA) on embedded charts. 3. **Self-Correction:** Feedback loops from user ratings to update Domain Taxonomy. --- ## Change History | Date | Version | Author | Changes | |------|---------|--------|---------| | 2026-02-03 | 3.0 | TextIQ | Initial PRD with hierarchical chunking and LlamaIndex | | 2026-02-06 | 3.1 | TextIQ | Updated to reflect dual-store architecture | | 2026-03-21 | 2.0 (Blueprint) | nguyetnta8 | C-REFRESH: Added LLM Provider Architecture (factory.py with Azure/vLLM dual-provider), Password-Protected Document Detection (password_detection.py), Tag Inheritance Engine (inheritance_engine.py), Multi-Lingual Translation Service (translation_service.py), Domain Term Extractor details, Tag Service details (TagManagementService, TagReviewService, TagGovernanceService), API route details for tags and admin_taxonomy, FR26. Added YAML frontmatter and TOC. | | 2026-04-10 | 2.1 | nguyennh25 | Added FR27-FR29: Domain Term Management via Airflow Plugin API (create, bulk create, update with cascade, delete with cascade). HMAC auth + RLS tenant isolation. | | 2026-05-06 | 2.2 | nguyennh25 | Re-aligned FR20-FR21 with the implemented chunk-tag endpoints (`PATCH /api/v1/blocks/{chunk_id}/tags/status`, `/bulk-status`, `/{tag_id}/name`, `DELETE /{tag_id}`); added FR30-FR33 (Document Upload & Approval — `/upload`, `/{id}/approve|reject`, presigned download/preview URLs, `DELETE /{id}`) and FR34 (Prometheus `/metrics` endpoint, HMAC-protected). | | 2026-05-06 | 2.3 | nguyennh25 | Three-part doc pass. **(a) src/api routes:** marked the legacy `src/api/{main.py, dependencies.py, middleware.py, routes/*}` surface as legacy code, deprecated — FR14 (`/documents`, `/query`) and FR23/FR24 (`/v1/admin/taxonomy/*`) now carry a "legacy code, deprecated" note pointing at the Airflow Plugin API equivalents (FR30-FR34). `src/api/schemas/*` remains live (shared data classes used by tagging pipeline). **(b) Legacy V1 capability:** marked the Excel "Symbol Resolution" extraction-strategy bullet as legacy V1 — not in active DAG; superseded by the V2 Docling pipeline + dual-store chunking. Aligns with the architecture-side dead-code pass that flagged `src/extraction/`, `src/adapters/legacy_to_milvus_adapter.py`, the orphaned `processing_tasks.py` tasks, and the `extracted_records` / `symbol_dictionaries` / `cross_references` PG tables as legacy V1 only. **(c) SeaweedFS clarification:** updated NFR9 to make explicit that SeaweedFS holds **both** uploaded source documents (Excel, PDF, etc.) and extracted images. | | 2026-06-08 | 2.6 | nguyennh25 | TEXTIQ-665: Excel Details flow now uses ks_xlsx_parser + markdownify `_FormattedConverter` instead of Docling. Dropped: Docling DocumentConverter, CellColorExtractor, sentinel injection (TEXTIQ-627 superseded for Details flow), `color_map`/`cell_values_map`/`cell_colors` from all chunk metadata. Kept: `_detect_and_extract_flowchart` (shapes→mermaid) + `collect_mermaid_blocks`/`restore_mermaid_blocks`. LibreOffice conversion timeout bumped to 300s. `parse_excel_file` returns one DTO per sheet (no service-side chunking). Updated FR3, FR9.3, FR9.4, Technical Constraints. Promoted `src/extraction_v2/ks_parser/` package. | | 2026-05-26 | 2.5 | nguyennh25 | TEXTIQ-625 md-branch follow-up: `.md` inputs now use the same mermaid-aware splitter as Excel (`chunk_with_atomic_mermaid(target=512, max_size=EMBEDDING_MODEL_CONTEXT_TOKENS - 400)`) so flowchart fences stay inline with surrounding prose, not orphaned as standalone leaves. Updated the Word/PDF/PPTX + Excel Detailed "Advanced Extraction Strategy" bullet (the `.md` sub-branch now matches Excel), FR9 (leaf-splitter description rewritten to "Excel and `.md` use `chunk_with_atomic_mermaid`; Word/PDF/PPTX use `SentenceSplitter`"), and FR9.2 (preservation guarantee now covers both Excel and `.md`). | | 2026-05-25 | 2.4 | nguyennh25 | Synced PRD with three implemented specs. **(a) TEXTIQ-625 (mermaid pipeline):** Excel flowcharts are now serialised to inline ` ```mermaid ` fences during pre-Docling OOXML processing — updated the Executive Summary problem statement (L46), the Word/PDF/PPTX and Excel Detailed "Advanced Extraction Strategy" bullets (L105/107/111), and FR5.1 (L148) to reflect mermaid embedding rather than NL flowchart description. **(b) TEXTIQ-625 (unified 2-level chunking):** FR9 (L154) rewritten from "3-level node hierarchy" to "2-level MarkdownNodeParser → branch-specific leaf splitter (Excel: `chunk_with_atomic_mermaid` target=512 / max≈7791; text: `SentenceSplitter(512, "\\n\\n")`)". Added FR9.2 (in-context mermaid preservation), FR9.3 (TEXTIQ-627 font-style ATX header injection), FR9.4 (4-layer embedding-budget defense: chunker cap, post-chunk audit, `EMBED_INCLUDE_KEYS` allow-list, parent-only `cell_colors`). **(c) TEXTIQ-627 (font-style ATX header injection):** validated by smoke run on the IF workbook (184 chunks, 90.2% non-trivial header_path, max 5274 tokens, 22 mermaid chunks preserved, 0 chunks exceed 8191 tokens). | --- _Updated 2026-06-08 -- TEXTIQ-665: Excel Details flow replaced with ks_xlsx_parser + markdownify; cell_colors/color_map dropped; sentinel injection superseded._