---
id: DE-ARCH
title: "textiq-doc-extraction Architecture"
version: "2.7"
status: draft
created: "2026-02-03"
lastUpdated: "2026-06-08"
author: nguyennh25
document_type: architecture
service: textiq-doc-extraction
---

# textiq-doc-extraction Architecture

## Table of Contents

- [Executive Summary](#executive-summary)
- [Decision Summary](#decision-summary)
- [Project Structure](#project-structure)
- [FR Category to Architecture Mapping](#fr-category-to-architecture-mapping)
- [Technology Stack Details](#technology-stack-details)
- [Document Processing Pipeline](#document-processing-pipeline)
- [Tagging & Governance Architecture (Phase 4 & 5)](#tagging--governance-architecture-phase-4--5)
- [LLM Provider Factory Architecture](#llm-provider-factory-architecture)
- [Extraction Module Inventory](#extraction-module-inventory)
- [Data Architecture](#data-architecture)
- [API Contracts](#api-contracts)
- [Implementation Patterns](#implementation-patterns)
- [Security Architecture](#security-architecture)
- [Performance Considerations](#performance-considerations)
- [Deployment Architecture](#deployment-architecture)
- [Architecture Decision Records (ADRs)](#architecture-decision-records-adrs)
- [Change History](#change-history)

---

## Executive Summary

textiq-doc-extraction is a Python-based document processing pipeline with REST API. The architecture prioritizes **extraction accuracy** over API complexity - the core innovation is the multi-pass LLM-powered extraction that preserves Excel semantics.

**Architecture Pattern:** Pipeline Architecture with API Gateway
**Primary Language:** Python 3.11+
**API Framework:** FastAPI
**Core Innovation:** Multi-pass extraction preserving 2D table structure

---

## Decision Summary

| Category | Decision | Version | Affects FRs | Rationale |
|----------|----------|---------|-------------|-----------|
| Language | Python | 3.11+ | All | Best ecosystem for data processing + LLM |
| Orchestration | Apache Airflow | 3.x | FR1-5 | Workflow orchestration, monitoring, retry logic |
| API Framework | FastAPI | 0.122 | FR29-33 | Async, type hints, auto docs |
| Database | PostgreSQL | 18 | FR18-21 | Structured store, JSON support, reliable |
| Vector Store | Milvus | 2.4 | FR19-20 | Open source, HNSW indexing, excellent Python SDK, advanced filtering |
| LLM Provider | Azure OpenAI + vLLM | GPT-4o | FR6-17, FR22-27 | Enterprise compliance + self-hosted option |
| Excel Processing (Large Table Records) | Docling | Latest | FR6-10 | Superior large table handling, HTML conversion |
| Excel Processing (Details) | ks_xlsx_parser + markdownify + openpyxl | vendored / Latest / 3.1.5 | FR1-10 | Form-layout-aware XLSX→HTML→MD (TEXTIQ-665); openpyxl for images, shapes, flowchart pre-processing |
| Chunking | LlamaIndex `MarkdownNodeParser` + mermaid-aware splitter | 0.12+ | FR6-10, FR18-21 | 2-level: markdown structure → 512-token leaves. Excel branch uses `chunk_with_atomic_mermaid` (target=512, max=`EMBEDDING_MODEL_CONTEXT_TOKENS - 400`); non-md/md branches use `SentenceSplitter(512, "\n\n")` (TEXTIQ-625) |
| Tokenization | tiktoken (cl100k_base) | Latest | FR6-10 | Token-accurate chunking compatible with text-embedding-3-large |
| Vector Indexing | LlamaIndex VectorStoreIndex | 0.12+ | FR18-21 | Manages Milvus + Postgres dual-store, leaf-only embedding |
| Object Storage | SeaweedFS (S3-compatible) | Latest | FR1-10 | Distributed blob storage — holds both uploaded source documents (Excel, PDF, etc.) and extracted images |
| Task Queue | Redis | 7+ | FR2, FR4 | Airflow backend, result caching |
| Containerization | Docker | 24+ | All | Consistent deployment |
| Package Manager | uv | Latest | All | Fast Python package management |

---

## Project Structure

```
textiq-doc-extraction/
├── docker-compose.yml          # Local dev environment
├── Dockerfile                  # Production container
├── pyproject.toml              # Dependencies (uv)
├── .env.example                # Environment template
├── README.md                   # Setup instructions
│
├── airflow/                    # Airflow orchestration layer
│   ├── dags/
│   │   ├── document_extraction_dag.py  # Main extraction DAG (~600 LOC)
│   │   ├── dag_config.py       # DAG configuration, timeouts, retry policy
│   │   └── tasks/              # External Python tasks
│   │       ├── parse_tasks.py        # Document parsing (Excel/Word/PDF/PPT)
│   │       ├── processing_tasks.py   # Chunking, embedding, storage tasks
│   │       │                         # ⚠️ Contains orphaned legacy code, deprecated
│   │       │                         # (not wired into the active DAG):
│   │       │                         #   - chunk_text_task
│   │       │                         #   - generate_embeddings_task
│   │       │                         #   - generate_embeddings_chunks
│   │       │                         #   - merge_pipeline_results
│   │       │                         #   - store_vectors_task
│   │       │                         #   - save_extracted_records_task
│   │       └── chunk_schema.py       # CHUNK_SCHEMA_VERSION stamped into chunk metadata
│   │
│   ├── plugins/                # Airflow plugin: FastAPI sub-apps + custom operators
│   │   ├── api/                # FastAPI sub-apps mounted on the api-server
│   │   │   ├── upload.py             # /api/v1/documents upload, approve, reject, presign, delete
│   │   │   ├── domain_terms.py       # /api/v1/domain-terms CRUD (single + bulk)
│   │   │   ├── tags.py               # /api/v1/blocks/{chunk_id}/tags status/bulk/rename/delete
│   │   │   ├── metrics.py            # /metrics Prometheus endpoint (HMAC-protected scrape)
│   │   │   ├── auth.py               # HMAC verification adapter over textiq_hmac
│   │   │   ├── middleware.py         # HMAC auth, correlation ID, request ID
│   │   │   ├── config.py             # Plugin configuration accessors
│   │   │   ├── db.py                 # SQLAlchemy engine (sync)
│   │   │   ├── milvus.py             # Milvus client helpers
│   │   │   ├── s3.py                 # SeaweedFS S3 helpers (circuit-breaker wrapped)
│   │   │   ├── dag.py                # Airflow DAG-trigger helper (Airflow 3.x compat)
│   │   │   ├── validation.py         # Extension + magic-byte + password-protect checks
│   │   │   ├── responses.py          # Standardized JSON responses
│   │   │   └── rls.py                # Row-Level Security connection manager
│   │   ├── operators/                # Lightweight custom Airflow operators
│   │   │   ├── validate_event.py     # ValidateEventOperator (webhook/API event)
│   │   │   └── fetch_document.py     # FetchDocumentOperator (SeaweedFS download)
│   │   ├── models/
│   │   │   └── xcom_schemas.py       # Dataclasses for XCom payloads.
│   │   │                             # ⚠️ Unused dataclasses (legacy code, deprecated;
│   │   │                             # only referenced by the orphaned legacy tasks above):
│   │   │                             #   - ParsedDocument, ContentChunks, ChunkEmbeddings
│   │   └── hooks/                    # (reserved; no custom hooks yet)
│   │
│   ├── config/                       # Tagging taxonomy assets
│   │   ├── tag_taxonomy.yaml
│   │   └── taxonomy/                 # standard_tags.json, namespaces.json
│   └── scripts/
│       └── create_pools.sh           # Bootstrap Airflow worker pools
│
├── docker/                           # Container build + compose assets
│   ├── Dockerfile.airflow            # Airflow image with uv-managed venv
│   ├── Dockerfile.airflow-proxy      # Proxy variant for restricted networks
│   ├── docker-compose.airflow.yml    # Airflow stack
│   └── docker-compose.yml            # Local dev stack (Postgres, Redis, Milvus, SeaweedFS)
│
├── src/
│   ├── __init__.py
│   │
│   ├── api/                    # FastAPI application — legacy code, partially deprecated
│   │   │                       # The REST handlers (`main.py`, `routes/*`,
│   │   │                       # `dependencies.py`, `middleware.py`) have
│   │   │                       # been superseded by the Airflow Plugin API
│   │   │                       # under `/api/v1/*` (see `airflow/plugins/api/`).
│   │   │                       # `schemas/` remains in active use as a shared
│   │   │                       # data-class library — imported by
│   │   │                       # `src/extraction_v2/{semantic_tagger,inheritance_engine}.py`
│   │   │                       # and `src/services/tag_*_service.py`.
│   │   ├── __init__.py
│   │   ├── main.py             # *(legacy code, deprecated)* FastAPI app entry
│   │   ├── dependencies.py     # *(legacy code, deprecated)*
│   │   ├── middleware.py       # *(legacy code, deprecated)*
│   │   ├── schemas/                 # *(still live — shared Pydantic schemas)*
│   │   │   ├── auth.py
│   │   │   ├── documents.py
│   │   │   ├── health.py            # used by `src/services/health.py`
│   │   │   ├── query.py             # used by `src/retrieval/*`
│   │   │   ├── symbols.py
│   │   │   └── tags.py              # used by tagging pipeline (extraction_v2 + services)
│   │   │
│   │   └── routes/             # *(legacy code, deprecated — see airflow/plugins/api/)*
│   │       ├── __init__.py
│   │       ├── documents.py    # superseded by airflow/plugins/api/upload.py
│   │       ├── query.py        # superseded by chatbot-service / MCP search
│   │       ├── health.py       # superseded by Airflow api-server health
│   │       ├── tags.py         # superseded by airflow/plugins/api/tags.py
│   │       └── admin_taxonomy.py  # superseded (governance moved to plugin scope)
│   │
│   ├── adapters/               # Legacy → V2 adapters — legacy code, deprecated
│   │   │                       # `legacy_to_milvus_adapter.py` is referenced only by the
│   │   │                       # dead `generate_embeddings_chunks` task in
│   │   │                       # `airflow/dags/tasks/processing_tasks.py`; not exercised
│   │   │                       # by the active DAG.
│   │   ├── __init__.py
│   │   └── legacy_to_milvus_adapter.py  # *(legacy code, deprecated)*
│   │
│   ├── core/                   # Business logic
│   │   ├── __init__.py
│   │   ├── config.py           # Settings management
│   │   └── exceptions.py       # Custom exceptions
│   │
│   ├── extraction/             # Legacy V1 pipeline — legacy code, deprecated
│   │   │                       # Not wired into the active DAG; superseded by
│   │   │                       # `extraction_v2/` (Docling-based V2 pipeline).
│   │   │                       # Symbol Resolution + Cross-Reference capabilities
│   │   │                       # implemented here are no longer exercised at runtime.
│   │   ├── __init__.py
│   │   ├── pipeline.py         # V1 extraction orchestrator *(unused)*
│   │   ├── excel_reader.py     # openpyxl wrapper *(unused)*
│   │   ├── table_detector.py   # Table boundary detection *(unused)*
│   │   ├── header_extractor.py # Multi-level header parsing *(unused)*
│   │   ├── merged_cell_resolver.py  # Merged cell propagation *(unused)*
│   │   ├── symbol_detector.py  # Symbol dictionary detection *(unused)*
│   │   ├── symbol_resolver.py  # Symbol → meaning resolution *(unused)*
│   │   ├── cross_ref_detector.py    # Cross-reference patterns *(unused)*
│   │   └── record_builder.py   # Structured record creation *(unused)*
│   │
│   ├── extraction_v2/          # Both flows (Large Table Records + Details)
│   │   ├── __init__.py
│   │   ├── pipeline.py         # V2 extraction orchestrator
│   │   ├── document_converter.py        # Docling wrapper
│   │   ├── large_table_extractor.py     # Large table detection and extraction
│   │   ├── html_table_extractor.py      # HTML table parsing
│   │   ├── llm_header_detector.py       # LLM-based header detection
│   │   ├── excel_parser_service.py      # Details flow: ks_xlsx_parser + markdownify (TEXTIQ-665)
│   │   ├── ks_parser/                   # ks_xlsx_parser integration (TEXTIQ-665)
│   │   │   ├── __init__.py              #   exports ksparser_to_sheets, ksparser_to_md
│   │   │   ├── pipeline.py             #   form-layout pipeline, _FormattedConverter, mermaid restore
│   │   │   └── mermaid.py              #   collect_mermaid_blocks, restore_mermaid_blocks
│   │   ├── excel_image_extraction_service.py  # Image extraction + AI descriptions
│   │   ├── shape_extractor.py           # Shape extraction and annotation
│   │   ├── azure_chat_service.py        # AI description generation (GPT-4o)
│   │   ├── domain_term_extractor.py     # Domain RAG term extraction
│   │   ├── semantic_chunker.py          # Text splitting
│   │   ├── semantic_tagger.py           # LLM-based semantic tag classification (Phase 3)
│   │   ├── tag_validator.py             # Tag validation against taxonomy (Phase 4)
│   │   ├── taxonomy_loader.py           # Taxonomy config loader
│   │   ├── inheritance_engine.py        # Tag inheritance: doc → section → block (Phase 3)
│   │   ├── password_detection.py        # Password-protected document detection
│   │   ├── translation_service.py       # LLM-powered term translation (EN/VI/JA)
│   │   └── record_builder.py   # Structured record creation
│   │
│   ├── services/               # Business services
│   │   ├── __init__.py
│   │   ├── auth_service.py              # API-key auth *(legacy code, deprecated — only used by `src/api/`)*
│   │   ├── azure_chat_service.py        # Azure chat wrapper
│   │   ├── batch_validator.py           # Batch upload validation *(legacy code, deprecated — only used by `src/api/`)*
│   │   ├── document_deletion_service.py # Doc deletion + cascade *(legacy code, deprecated — only used by `src/api/`; pulls in legacy `knowledge.{vector_store,structured_store}`)*
│   │   ├── document_service.py          # Document CRUD *(legacy code, deprecated — only used by `src/api/`)*
│   │   ├── file_storage.py              # File storage abstraction *(legacy code, deprecated — only used by `src/api/`)*
│   │   ├── file_validator.py            # MIME / extension validation *(legacy code, deprecated — only used by `src/api/`)*
│   │   ├── health.py                    # Dependency health checks *(legacy code, deprecated — only used by `src/api/`)*
│   │   ├── pipeline_events.py           # Redis pub/sub stage events
│   │   ├── progress_tracker.py          # Upload progress tracker *(legacy code, deprecated — only used by `src/api/`)*
│   │   ├── rate_limiter.py              # API-key rate limiter *(legacy code, deprecated — only used by `src/api/`)*
│   │   ├── seaweedfs_service.py         # SeaweedFS S3 client (circuit-breaker wrapped)
│   │   ├── symbol_service.py            # Symbol service *(legacy code, deprecated — only used by `src/api/`)*
│   │   ├── tag_governance_service.py    # Trending analysis & promotion *(legacy code, deprecated — only used by `src/api/`)*
│   │   ├── tag_management_service.py    # Tag CRUD operations *(legacy code, deprecated — only used by `src/api/`)*
│   │   └── tag_review_service.py        # HITL review workflow *(legacy code, deprecated — only used by `src/api/`)*
│   │
│   ├── knowledge/              # Knowledge base management
│   │   ├── __init__.py
│   │   ├── domain_dictionary_service.py # Domain term management
│   │   ├── embeddings.py                # *(legacy code, deprecated — only orphan `generate_embeddings_task`)*
│   │   ├── vector_store.py              # *(legacy code, deprecated — only orphan `store_vectors_task`)*
│   │   ├── structured_store.py          # *(legacy code, deprecated — only orphan `save_extracted_records_task`)*
│   │   ├── domain_aware_retriever.py    # *(legacy code, deprecated — no live consumer)*
│   │   ├── query_term_matcher.py        # *(legacy code, deprecated — only used by `domain_aware_retriever`)*
│   │   ├── term_index.py                # *(legacy code, deprecated — only used by `query_term_matcher`)*
│   │   ├── indexer.py                   # *(legacy code, deprecated — zero importers anywhere)*
│   │   ├── metadata_service.py          # *(legacy code, deprecated — zero importers anywhere)*
│   │   └── query_helper.py              # *(legacy code, deprecated — zero importers anywhere)*
│   │
│   ├── retrieval/              # Query processing — legacy code, deprecated
│   │   │                       # Both files are reachable only via the deprecated
│   │   │                       # `src/api/routes/query.py`; `semantic_retriever`
│   │   │                       # additionally pulls in the legacy `knowledge.embeddings`
│   │   │                       # stack. Active query path is chatbot-service / MCP.
│   │   ├── __init__.py
│   │   ├── answer_generator.py          # *(legacy code, deprecated)*
│   │   └── semantic_retriever.py        # *(legacy code, deprecated)*
│   │
│   ├── qa/                     # *(empty — only `__init__.py`; reserved for future use)*
│   │   └── __init__.py
│   │
│   ├── llm/                    # LLM integration (provider-agnostic)
│   │   ├── __init__.py
│   │   ├── factory.py          # Provider factory (Azure/vLLM)
│   │   └── chat_service.py     # LangChain ChatService wrapper
│   │
│   ├── models/                 # *(empty — only `__init__.py`; reserved for future use)*
│   │   └── __init__.py
│   │
│   └── db/                     # Database layer
│       ├── __init__.py
│       ├── session.py          # SQLAlchemy session
│       ├── models.py           # ORM models
│       └── migrations/         # Alembic migrations
│
├── tests/
│   ├── __init__.py
│   ├── conftest.py             # Pytest fixtures
│   ├── test_extraction/        # Extraction tests
│   ├── test_knowledge/         # Knowledge base tests
│   ├── test_retrieval/         # Retrieval tests
│   ├── test_qa/                # Q&A tests
│   └── test_api/               # API integration tests
│
└── scripts/
    ├── seed_db.py              # Database seeding
    ├── process_batch.py        # Batch processing CLI
    └── evaluate_accuracy.py    # Q&A accuracy evaluation
```

---

## FR Category to Architecture Mapping

| FR Category | Primary Module | Supporting Modules |
|-------------|----------------|-------------------|
| Document Management (FR1-5) | `api/routes/documents.py` | `airflow/dags/document_extraction_dag.py`, `airflow/plugins/operators/`, `airflow/plugins/api/upload.py` |
| Table Extraction (FR6-10) | `extraction_v2/` (Both flows) | `airflow/dags/tasks/parse_tasks.py`, `llm/` for AI descriptions |
| Symbol Resolution (FR11-14) — *legacy code, deprecated (V1, not in active DAG)* | `extraction/symbol_*.py` *(legacy code, deprecated)* | `knowledge/structured_store.py` *(legacy code, deprecated — write path dead)* |
| Cross-Reference (FR15-17) — *legacy code, deprecated (V1, not in active DAG)* | `extraction/cross_ref_detector.py` *(legacy code, deprecated)* | `knowledge/structured_store.py` *(legacy code, deprecated — write path dead)* |
| Knowledge Base (FR18-21) | `knowledge/` | `db/`, `airflow/dags/tasks/processing_tasks.py` |
| Q&A Processing (FR22-28) | `qa/`, `retrieval/` | `llm/` |
| API Access (FR29-34) | `airflow/plugins/api/` (active) | `src/api/` *(legacy code, deprecated)* |
| Orchestration & Workflow | `airflow/dags/` | `airflow/dags/dag_config.py`, `airflow/dags/tasks/chunk_schema.py` |
| Tag Management (FR20-22) | `airflow/plugins/api/tags.py` | `extraction_v2/tag_validator.py`, `extraction_v2/taxonomy_loader.py` (`src/api/routes/tags.py` *legacy code, deprecated*) |
| Tag Governance (FR23-25) | `services/tag_governance_service.py` | `extraction_v2/translation_service.py` (`src/api/routes/admin_taxonomy.py` *legacy code, deprecated*) |
| Password Detection (FR26) | `extraction_v2/password_detection.py` | `extraction_v2/pipeline.py` |

---

## Technology Stack Details

### Core Technologies

**Python 3.11+**
- Type hints throughout
- Async/await for I/O operations
- Dataclasses for models

**Apache Airflow 3.x**
- DAG-based workflow orchestration
- Task dependency management
- Built-in monitoring and alerting
- SLA tracking (2-hour default)
- External Python decorator (`@task.external_python`) for isolated task execution
- Failure callbacks and retry logic

**FastAPI**
- Async request handling
- Automatic OpenAPI documentation
- Pydantic validation
- Dependency injection

**PostgreSQL 18**
- Primary structured data store
- JSONB columns for flexible record storage
- Full-text search capability (fallback)
- Source metadata indexing

**Milvus 2.4**
- Vector similarity search with HNSW/IVF indexing
- Scalar field filtering (file, sheet, record_id, domain_category)
- Local deployment via Docker for development
- Milvus Standalone or Cluster for production
- Native support for hybrid search (dense + sparse vectors)

**Redis 7+**
- Airflow backend for task queue management
- Result caching
- Distributed locking

**Docling (Latest)**
- Excel to HTML conversion with table preservation
- Superior handling of complex table layouts and large tables
- Primary extraction approach for Large Table Records flow
- Preserves table structure and merged cells

**openpyxl 3.1.5**
- Pre-processing for Details flow (image/shape extraction, font-style ATX header injection, per-graph mermaid embedding)
- Full .xlsx/.xls/.xlsm support
- Image extraction with position tracking
- Shape extraction and cell annotation application
- Pre-Docling OOXML walk (`xl/styles.xml` + `sharedStrings.xml` + `worksheets/sheetN.xml`) to materialise font-style headers and serialise flowchart connectors as inline mermaid blocks before parsing (TEXTIQ-625, TEXTIQ-627)
- Comment processing

**SeaweedFS**
- Distributed S3-compatible object store for both uploaded source documents (Excel, PDF, etc.) and extracted images
- Fast upload, retrieval, and presigned-URL access for documents and images
- Horizontal scalability

**BeautifulSoup4 + lxml**
- HTML parsing for Docling output
- Fast HTML table extraction
- Supports complex HTML structures

**LlamaIndex Core + Integrations**
- `MarkdownNodeParser` for the parent (section) level of the 2-level chunk hierarchy
- `SentenceSplitter` (text branch) and `chunk_with_atomic_mermaid` (Excel branch) for the leaf level
- `VectorStoreIndex` for embedding and indexing leaf nodes
- `StorageContext` for unified dual-store management
- `AutoMergingRetriever` for parent-chain context expansion at query time
- `MilvusVectorStore` integration for vector storage (leaves only)
- `PostgresDocumentStore` integration for the full node hierarchy
- `AzureOpenAIEmbedding` (text-embedding-3-large, dim=3072)

**tiktoken**
- Token-accurate text splitting using `cl100k_base` encoding
- Compatible with OpenAI embedding models
- Used by `SentenceSplitter` (text branch) and the `chunk_with_atomic_mermaid` line-based splitter (Excel branch)

**uv**
- Fast Python package installer and resolver
- Virtual environment management in Airflow containers
- All heavy dependencies installed in `/opt/airflow/.venv`

### Integration Points

```
┌─────────────────────────────────────────────────────────────────────┐
│                            CLIENT                                    │
└─────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         FastAPI (api/)                               │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐                  │
│  │  /documents │  │   /query    │  │   /health   │                  │
│  └─────────────┘  └─────────────┘  └─────────────┘                  │
│  ┌──────────────────────┐  ┌──────────────────────┐                  │
│  │ /v1/blocks/*/tags    │  │ /v1/admin/taxonomy   │                  │
│  └──────────────────────┘  └──────────────────────┘                  │
└─────────────────────────────────────────────────────────────────────┘
                     │
                     ▼
┌──────────────────────────────────────────────────────────────────┐
│      Apache Airflow (airflow/)                                   │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │ Document Extraction DAG                                    │  │
│  │                                                            │  │
│  │ validate → fetch → save_metadata → detect_format (branch)  │  │
│  │                                                            │  │
│  │ ┌──────────────────────────────────────────────────────┐   │  │
│  │ │ EXCEL LARGE TABLE PIPELINE (Flat Records)            │   │  │
│  │ │ parse_excel_large_tables → detect_headers            │   │  │
│  │ │ → chunk_large_table_records → convert_to_nl          │   │  │
│  │ │ → extract_domain_terms → tag_records                 │   │  │
│  │ │ → store_large_table_to_llamaindex (Milvus only)      │   │  │
│  │ └──────────────────────────────────────────────────────┘   │  │
│  │                                                            │  │
│  │ ┌──────────────────────────────────────────────────────┐   │  │
│  │ │ EXCEL DETAILED PIPELINE (2-level mermaid-aware)      │   │  │
│  │ │ parse_excel_detailed → upload_images_to_seaweedfs    │   │  │
│  │ │ → chunk_excel_detailed                               │   │  │
│  │ │   (markdown → chunk_with_atomic_mermaid              │   │  │
│  │ │    target=512, max=embed_ctx-400 ≈ 7791)             │   │  │
│  │ │ → extract_domain_terms_nodes → tag_records_nodes     │   │  │
│  │ │ → store_to_llamaindex (Milvus + PostgresDocStore)    │   │  │
│  │ └──────────────────────────────────────────────────────┘   │  │
│  │                                                            │  │
│  │ ┌──────────────────────────────────────────────────────┐   │  │
│  │ │ TEXT PIPELINE - Word/PDF/PPT (2-level)               │   │  │
│  │ │ parse_{word|pdf|pptx} → text_parse_result            │   │  │
│  │ │ → chunk_text                                         │   │  │
│  │ │   (markdown → SentenceSplitter(512, "\\n\\n"))         │   │  │
│  │ │ → extract_domain_terms_nodes → tag_records_nodes     │   │  │
│  │ │ → store_to_llamaindex (Milvus + PostgresDocStore)    │   │  │
│  │ └──────────────────────────────────────────────────────┘   │  │
│  │                                                            │  │
│  │ [store_large_tables, store_excel_detailed, store_text]     │  │
│  │   → update_document_status → cleanup                      │  │
│  └────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────┘
         │              │              │
         ▼              ▼              ▼
┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│   Redis     │  │ PostgreSQL  │  │Azure OpenAI │
│  (Airflow   │  │ (Airflow    │  │  / vLLM     │
│   Queue)    │  │  Metadata)  │  │  (LLM)      │
└─────────────┘  └─────────────┘  └─────────────┘
                                         │
          ┌──────────────────────────────┼──────────────────────┐
          ▼                              ▼                      ▼
 ┌──────────────────┐          ┌─────────────┐        ┌─────────────┐
 │ PostgreSQL       │          │   Milvus    │        │ SeaweedFS   │
 │ (Structured +    │          │  (Vector)   │        │  (Images)   │
 │  DocStore for    │          │ Leaf nodes  │        └─────────────┘
 │  all hierarchy   │          │ only (3072d)│
 │  nodes via       │          └─────────────┘
 │  LlamaIndex)     │
 └──────────────────┘
```

---

## Document Processing Pipeline

### Extraction Lifecycle Flow

```mermaid
graph TD
    subgraph Input
        Upload[Upload Document]
    end

    subgraph Orchestration [Airflow DAG]
        Validate[Validate Event] --> Fetch[Fetch Document]
        Fetch --> SaveMeta[Save Document Metadata]
        SaveMeta --> Parse{File Type?}

        Parse -- Excel --> ParallelStart(( ))
        Parse -- Word/PDF/PPT --> BranchText[Text Pipeline]
        Parse -- Unsupported --> Unsupported[Unsupported Format]

        subgraph LargeTableFlow [Excel Large Table Pipeline - Flat Records]
            ParallelStart --> |Flow 1| LargeTable[parse_excel_large_tables]
            LargeTable --> DetectHeader[detect_headers]
            DetectHeader --> ChunkRows[chunk_large_table_records]
            ChunkRows --> NL[convert_to_natural_language]
            NL --> DomainFlat[extract_domain_terms]
            DomainFlat --> TagFlat[tag_records]
            TagFlat --> StoreLargeTable[store_large_table_to_llamaindex<br/>Milvus only]
        end

        subgraph DetailedFlow [Excel Detailed Pipeline - Hierarchical]
            ParallelStart --> |Flow 2| Details[parse_excel_detailed]
            Details --> UploadImages[upload_images_to_seaweedfs]
            UploadImages --> HierChunkExcel[chunk_excel_detailed<br/>MarkdownNodeParser → chunk_with_atomic_mermaid<br/>target=512 / max≈7791 tokens]
            HierChunkExcel --> DomainNodes1[extract_domain_terms_nodes]
            DomainNodes1 --> TagNodes1[tag_records_nodes]
            TagNodes1 --> StoreDetailed[store_to_llamaindex<br/>Milvus + PostgresDocStore]
        end

        subgraph TextFlow [Text Pipeline - Hierarchical]
            BranchText --> ParseWord[parse_word]
            BranchText --> ParsePDF[parse_pdf]
            BranchText --> ParsePPT[parse_powerpoint]
            ParseWord --> JoinText[text_parse_result]
            ParsePDF --> JoinText
            ParsePPT --> JoinText
            JoinText --> HierChunkText[chunk_text<br/>MarkdownNodeParser → SentenceSplitter(512, "\\n\\n")]
            HierChunkText --> DomainNodes2[extract_domain_terms_nodes]
            DomainNodes2 --> TagNodes2[tag_records_nodes]
            TagNodes2 --> StoreText[store_to_llamaindex<br/>Milvus + PostgresDocStore]
        end

        StoreLargeTable --> UpdateStatus[update_document_status]
        StoreDetailed --> UpdateStatus
        StoreText --> UpdateStatus
        Unsupported --> Cleanup[cleanup]
        UpdateStatus --> Cleanup
    end

    Cleanup --> Complete((End))
```

### Excel Hybrid Pipeline Architecture

For Excel documents, the system employs a **dual-processing strategy** that runs two extraction flows in parallel, then merges their results:

#### Large Table Records Flow (Docling-based, Flat)

**Purpose:** Efficiently extract large, structured tables while preserving layout. Uses flat records (1 row = 1 Document), **not** hierarchical chunking.

**Flow:**
1. **parse_excel_large_tables** - Convert Excel to HTML using Docling
2. **detect_headers** - LLM-based multi-level header detection
3. **chunk_large_table_records** - Split large tables into row-level records
4. **convert_large_tables_to_natural_language** - Convert structured rows to readable text
5. **extract_domain_terms** - Extract domain-specific terminology (flat record variant)
6. **tag_records** - Assign semantic tags (flat record variant)
7. **store_large_table_to_llamaindex** - Store to Milvus only via `VectorStoreIndex.from_documents()`

**Storage:** Milvus only (no PostgresDocumentStore) -- each row is an independent Document with full record JSON in metadata.

#### Details Flow (Docling + openpyxl hybrid, 2-level Mermaid-Aware)

**Purpose:** Extract images, shapes, comments, and text content while preserving flowcharts as in-context mermaid blocks. Chunking is now a 2-level shape (markdown structure → 512-token leaves) with explicit defenses against the embedding-model context window.

**Flow:**
1. **parse_excel_detailed** - Form-layout-aware extraction using ks_xlsx_parser (TEXTIQ-665). Pre-parse `_detect_and_extract_flowchart` serialises OOXML drawing shapes to mermaid text in cells (TEXTIQ-625 Goal A); `collect_mermaid_blocks` captures those fences before the parse loop. `ks_xlsx_parser` converts the workbook to chunked HTML with form-layout semantics (merged cells, column ordering, page-break reordering); `_FormattedConverter` (markdownify subclass) converts HTML→MD, preserving inline bold (`**`), strikethrough (`~~`), and `<span style="color:...">`. `restore_mermaid_blocks` re-embeds mermaid fences in the final MD. Returns one markdown section per sheet (no service-side chunking). `flowchart_chunks` is preserved as `[]` for backwards-compatible downstream consumers.
2. **upload_images_to_seaweedfs** - Store extracted images in blob storage, replace base64 with URLs
3. **chunk_excel_detailed** - 2-level LlamaIndex `MarkdownNodeParser` → `chunk_with_atomic_mermaid(target=512, max_size=embed_max_tokens-400)` (TEXTIQ-625 Goal B). Mermaid fences are kept atomic when they fit the cap; oversized fences fall through to `split_oversize_mermaid` on body lines only. Backward tail-merge below `MIN_CHUNK_TOKENS=64` and `prune_header_only_nodes` (regex `^\s*#{1,6} [^\n]+\s*$`) suppress fragments and header-only leaves.
4. **extract_domain_terms_nodes** - Extract domain terms from the resulting nodes
5. **tag_records_nodes** - Assign semantic tags
6. **store_to_llamaindex** - Dual-store: all nodes -> PostgresDocumentStore, leaf nodes -> Milvus (parent-only `cell_colors` to stay under Milvus's 64KB dynamic-field envelope)

#### Independent Storage Paths

Each pipeline stores results independently -- there is no merge step:
- **Large Table pipeline** -> `store_large_table_to_llamaindex` (Milvus only)
- **Excel Detailed pipeline** -> `store_to_llamaindex` (Milvus + PostgresDocumentStore)
- **Text pipeline** -> `store_to_llamaindex` (Milvus + PostgresDocumentStore)

**Trigger Rule:** Default `all_success` on chunking, domain-term, tag, and store tasks so a skipped branch (e.g. text path when branch picks Excel) cascades a skip through its sub-pipeline instead of running with empty inputs. `select_parse_result` keeps `none_failed_min_one_success` because its three direct upstreams are mutually-exclusive skips.

### Text Document Pipeline (Word/PDF/PowerPoint) -- 2-Level

**Flow:**
1. **parse_{word|pdf|powerpoint}** - Format-specific parsing via Docling, producing page-based text
2. **text_parse_result** - Join point merging all text format outputs
3. **chunk_text** - 2-level LlamaIndex `MarkdownNodeParser` → leaf splitter (TEXTIQ-625, collapsed from the legacy 3-level shape on 2026-05-25). The non-md sub-branch (Word/PDF/PPTX) uses `SentenceSplitter(chunk_size=512, paragraph_separator="\n\n")`. The .md sub-branch shares the **Excel parser map** (`MermaidAwareLineSplitter(target=512, max=embed_max_tokens-400)`) so ```mermaid fences stay inline with their surrounding prose as atomic line-groups; oversize single fences fall through to `split_oversize_mermaid` on body lines only (TEXTIQ-625 follow-up, 2026-05-26).
4. **extract_domain_terms_nodes** - Extract domain_category and domain_terms for each node
5. **tag_records_nodes** - Assign semantic tags (req:*, risk:*, info:*, topic:*) to nodes
6. **store_to_llamaindex** - Dual-store: All nodes -> PostgresDocumentStore, Leaf nodes -> Milvus

**Node Hierarchy:**
| Level | Chunk Size | Role |
|-------|-----------|------|
| 0 (Root) | markdown section (variable) | High-level context for AutoMergingRetriever |
| 1 (Leaf) | 512 tokens | Embedded in Milvus, retrieved first |

### Hierarchical Chunking Architecture

> **2026-05-25 (TEXTIQ-625):** Both branches were collapsed from a 3-level `HierarchicalNodeParser` shape to a 2-level `MarkdownNodeParser → leaf-splitter` shape. AutoMergingRetriever still operates over the parent chain stored in PostgresDocumentStore; only Milvus stores the leaves.

The system uses LlamaIndex's `MarkdownNodeParser` for the first level (markdown structure → section nodes) and a branch-specific leaf splitter for the second. The Excel branch's leaf splitter is mermaid-aware so flowchart diagrams stay atomic.

#### Chunk Sizes by Pipeline

| Pipeline | Level 0 (Parent) | Level 1 (Leaf) | Leaf Splitter | Rationale |
|----------|------------------|----------------|----------------|-----------|
| Text — Word/PDF/PPTX | markdown section (variable) | 512 tokens | `SentenceSplitter(chunk_size=512, paragraph_separator="\n\n")` | Prose benefits from paragraph-aligned splitting at a 512-token target |
| Text — `.md` files | markdown section (variable) | target 512 / max ≈ 7791 tokens | `chunk_with_atomic_mermaid(target=512, max_size=embed_max_tokens-400)` | Same mermaid-aware splitter as Excel — keeps ```mermaid fences inline with their explanatory prose; only fences > chunker_max trigger body-line splits |
| Excel Detailed | markdown section (variable) | target 512 / max ≈ 7791 tokens | `chunk_with_atomic_mermaid(target=512, max_size=embed_max_tokens-400)` | Mermaid blocks stay atomic; tables and sheet sub-trees fit into the embedding context window with a 400-token safety buffer |

#### Dual-Store Pattern

```
┌──────────────────────────────────────────────────────────────┐
│                  LlamaIndex Storage Layer                     │
├───────────────────────────┬──────────────────────────────────┤
│  PostgresDocumentStore    │  MilvusVectorStore               │
│  ─────────────────────    │  ─────────────────               │
│  • ALL nodes (all levels) │  • Leaf nodes only               │
│  • Full node hierarchy    │  • 3072-dim vectors              │
│  • Parent-child refs      │  • text-embedding-3-large        │
│  • Node traversal         │  • COSINE similarity             │
│  • Table: hierarchical_   │  • Fast ANN search               │
│    nodes (JSONB)          │                                  │
│  • Namespace: hierarchical│                                  │
├───────────────────────────┴──────────────────────────────────┤
│  StorageContext.from_defaults(                                │
│      vector_store=milvus_store,                              │
│      docstore=postgres_docstore                              │
│  )                                                           │
└──────────────────────────────────────────────────────────────┘
```

#### Metadata Enrichment Pipeline

| Stage | Metadata Added | Node Fields |
|-------|---------------|-------------|
| Parsing | document_id, filename, page_number, document_type | Base identity |
| Hierarchical Chunking | token_count, char_count | Size metrics |
| Domain Extraction | domain_category, domain_terms | Domain filtering |
| Tagging | tags (req:*, risk:*, info:*, topic:*) | Semantic classification |

#### Embedding Configuration

| Parameter | Value |
|-----------|-------|
| Model | Azure OpenAI text-embedding-3-large (or vLLM equivalent) |
| Dimensions | 3072 |
| Batch size | 16 (max 100 per API call) |
| Encoding | tiktoken cl100k_base |
| Metric | COSINE |
| `EMBEDDING_MODEL_CONTEXT_TOKENS` | 8191 (text-embedding-3-large hard limit) |
| `EMBED_SAFETY_BUFFER` | 400 |
| Effective chunker cap (`chunker_max`) | 7791 tokens (8191 - 400) |

### Mermaid-Aware Chunking & Font-Style Header Injection (TEXTIQ-625, TEXTIQ-627)

> **Status:** implemented 2026-05-22 (TEXTIQ-627 production wire-up) and 2026-05-25 (TEXTIQ-625 unified 2-level shape). Smoke runs pass on the `IF` and `overview` workbooks (184 chunks at 90.2% header coverage, 0 chunks > 8191 tokens, 22 mermaid blocks preserved).

**Goal A — Per-graph mermaid embedding (TEXTIQ-625):** The Excel pre-processing step walks `xl/drawings/drawing*.xml` and the connector/shape graph in OOXML, serialises each flowchart to a `mermaid flowchart TD` block, and writes the fenced block back into the originating worksheet cell **before** ks_xlsx_parser parses (TEXTIQ-665). `collect_mermaid_blocks` captures those fences once before the per-sheet loop; `restore_mermaid_blocks` re-embeds them after markdownify conversion. The legacy `FlowchartDetector` / `FlowchartExtractor` modules and their three dev scripts were deleted; `flowchart_chunks` is retained as an always-`[]` key on the parse result for downstream callers.

**Goal B — Mermaid-aware line-based chunker (TEXTIQ-625):** The Excel branch **and** the .md sub-branch of the text pipeline share the same leaf splitter: `chunk_with_atomic_mermaid(text, target=512, max_size=chunker_max, tokenize=tiktoken.cl100k_base)` (`src/extraction_v2/mermaid_chunking.py`), wired in via `build_excel_parser_map`. It walks the markdown line-by-line, keeps each ` ```mermaid ` … ` ``` ` fence atomic when it fits the cap so the diagram stays adjacent to its explanatory prose, and only falls through to `split_oversize_mermaid` on body lines (never on the fence boundary). A `MIN_CHUNK_TOKENS=64` backward tail-merge collapses trailing fragments into their predecessor, and `prune_header_only_nodes` (regex `^\s*#{1,6} [^\n]+\s*$`) drops nodes whose entire payload is a single ATX heading. The non-md text sub-branch (Word/PDF/PPTX) still uses the stock `SentenceSplitter(512, "\n\n")` because Docling's prose markdown has reliable `\n\n` paragraph anchors and rarely emits mermaid.

**Font-style ATX header injection (TEXTIQ-627):** For Excel, openpyxl walks `xl/styles.xml` + `sharedStrings.xml` + `worksheets/sheetN.xml` and ranks bold font sizes per workbook. `_detect_body_size` picks the mode of non-bold sizes as the body baseline; the largest bold size becomes H1 (overridden by sheet name for the implicit top-level header), and the next five descending bold sizes map to H2..H6. Cells are wrapped with the `§§HN§§ … §§/HN§§` sentinel scheme before Docling sees them, then sentinels are rewritten as ATX `#` headings in the markdown. This is what lets `MarkdownNodeParser` produce a meaningful section tree on workbooks whose authors used font sizes (not Word-style headings) to mark structure.

**4-layer embedding-budget defense:**
1. **Chunker cap** — `chunker_max = embedding_model_context_tokens - 400 = 7791` is the hard upper bound on every Excel leaf at the chunking stage. The md branch's orphan-atomic mermaid post-processor uses the full 8191 (no buffer) via `_split_fence(raw, max_tokens=embedding_model_context_tokens)`.
2. **Post-chunk audit** — per-leaf `MetadataMode.EMBED` token count is logged; chunks exceeding `embedding_model_context_tokens` emit a WARNING (`airflow/dags/tasks/processing_tasks.py` L2885-2906) so any regression is visible without crashing the DAG.
3. **`EMBED_INCLUDE_KEYS` allow-list** — only `{filename, sheet_name, document_type, element_type}` are merged into the text envelope that's sent to the embedding API, preventing noisy metadata (e.g. `cell_colors`, `tags`, `domain_terms`) from inflating the token count.
4. **`cell_colors` removed (TEXTIQ-665)** — colour metadata (`cell_colors`, `color_map`, `cell_values_map`) is fully removed from all chunk metadata. The 64KB Milvus dynamic-field envelope cap is met purely via the `EMBED_INCLUDE_KEYS` allow-list in point (c).

### Airflow DAG Orchestration

**Key Features:**
- **External Python Tasks** - Heavy processing runs in isolated `/opt/airflow/.venv` environment
- **Parallel Execution** - Excel Large Table and Detailed pipelines run concurrently
- **Independent Storage** - Each pipeline has its own storage task (no merge step)
- **Failure Handling** - Callbacks for debugging
- **Retry Policy** - Default `retries=0, retry_delay=30s`. Only the five format parsers (`parse_excel_large_tables`, `parse_excel_detailed`, `parse_word_document`, `parse_pdf_document`, `parse_powerpoint_document`) set `retries=2` with 30s exponential backoff, because fetch+parse is the only step with transient IO flakiness worth retrying.
- **SLA Tracking** - 2-hour SLA per task with miss callbacks
- **Cleanup** - Automatic temp file cleanup regardless of success/failure
- **Pickle File Handoffs** - Large data transferred via pickle files to avoid XCom 48KB limit

---

## Pipeline Events (DE → UPA)

Browsers need live progress updates for extraction runs. DE does not run a Socket.IO server; instead it publishes stage-transition events to Redis pub/sub, and the User Prompt API (UPA) service relays them to the browser over its Socket.IO `/pipeline` namespace.

**Channel:** `{prefix}:{tenant_id}:{node_id}` (default prefix `pipeline`). The channel is routed by tenant + node so UPA can subscribe with the pattern `pipeline:<tenant_id>:*` and scope events per browser session.

**Event schema (JSON):**

| Field | Type | Notes |
|-------|------|-------|
| `correlation_id` | string | Airflow `dag_run_id`. Groups events from one extraction run. |
| `document_id` | UUID string | Target document. |
| `tenant_id` | UUID string | Routing + authorization scope. |
| `node_id` | UUID string | Routing. |
| `stage` | enum | One of `pending`, `processing`, `parsing`, `chunking`, `indexing`, `completed`, `failed`. Aligned with `documents.status` vocabulary; `parsing`/`chunking`/`indexing` are event-only sub-phases of DB `processing`. |
| `status` | enum | One of `started`, `succeeded`, `failed`. Mid-pipeline stages always emit `started`; terminal stages emit `{completed, succeeded}` or `{failed, failed}`. |
| `timestamp` | string | ISO-8601 UTC with `Z` suffix. |
| `error` | string (optional) | Present only on `{failed, failed}`; short human-readable reason (no stack trace). |

**Emit boundaries:** `save_document_metadata_task` → `pending` then `processing`; each parser → `parsing`; `chunk_large_table_records` → `chunking`; each `store_*_llamaindex_task` → `indexing`; `update_document_status_task` → `completed` or `failed`, resolved by the upstream `resolve_terminal_outcome` task which inspects all TI states via Airflow context.

**Delivery semantics:** Redis pub/sub is fire-and-forget and **not persisted**. If no subscriber is connected at publish time, the message is dropped. The source of truth for document lifecycle remains `documents.status` in Postgres; events are a live UX signal only. If UPA is offline during a run, the browser falls back to polling `GET /documents/{id}` for terminal state.

**Feature flag:** Controlled by `PIPELINE_EVENTS_ENABLED` (default `false`). When disabled, the publisher short-circuits without opening a Redis connection. Redis publish failures are logged at WARNING and swallowed — emit failures never fail a DAG task.

**What is never emitted:** document content, parsed text, filenames, domain terms, tag values, stack traces. Events carry routing IDs and status only.

---

## Tagging & Governance Architecture (Phase 4 & 5)

> **C-REFRESH 2026-03-21:** Expanded with service implementation details from code analysis.

### Tag Services (3 new services)

**TagManagementService** (`src/services/tag_management_service.py`):
- Handles CRUD operations for tags on extracted records.
- Enforces validation via `TagValidator` against Standard Taxonomy loaded by `TaxonomyLoader`.
- Manages "Shadow Tags" (dynamic) vs "Verified Tags" (pre-defined).
- Add/edit operations validate tag name, check against taxonomy, auto-complete from pre-defined list.
- Tags stored as JSONB array in `extracted_records.tags` column.

**TagReviewService** (`src/services/tag_review_service.py`):
- Implements Human-in-the-Loop (HITL) workflow.
- Manages "Pending" state for new tags via `review_tag()` method.
- Supports Confirm/Reject decisions with `TagReviewDecision` enum.
- Logs review history (Approver, Timestamp, Decision) via `TagHistoryEntry`.

**TagGovernanceService** (`src/services/tag_governance_service.py`):
- **DynamicTagAnalyzer:** Analyzes tag usage frequency and distribution (trending tags). SQL query aggregates from JSONB `tags` column with configurable thresholds (min_usage=50, min_documents=3) and namespace filtering.
- **TagPromotionService:** Manages promotion of dynamic tags to Standard Taxonomy. Batch-updates all records renaming old tag to new canonical tag. Optionally writes to taxonomy config file.
- Implements "Shadow Taxonomy" concept for emerging terms.

### API Routes

> **Legacy code, deprecated:** The `src/api/routes/*` FastAPI surface is no longer the runtime entry point. All endpoints below have been superseded by the Airflow Plugin API (see the four "Airflow Plugin API — *" subsections that follow). The files remain in-tree for reference until the cleanup PR lands.

**Tag Management** (`src/api/routes/tags.py` — *legacy code, deprecated*, superseded by `airflow/plugins/api/tags.py`):
- ~~`GET /v1/blocks/{record_id}/tags`~~ -- Get all tags for a record
- ~~`POST /v1/blocks/{record_id}/tags`~~ -- Add or edit a tag
- ~~`POST /v1/blocks/{record_id}/tags/review`~~ -- Review pending tag (HITL)

**Admin Taxonomy** (`src/api/routes/admin_taxonomy.py` — *legacy code, deprecated*, governance moved to plugin scope):
- ~~`GET /v1/admin/taxonomy/trending`~~ -- Get trending dynamic tags (with min_usage, min_documents, namespace filters)
- ~~`GET /v1/admin/taxonomy/stats`~~ -- Get tag usage statistics (by namespace, by status, dynamic count)
- ~~`POST /v1/admin/taxonomy/promote`~~ -- Promote dynamic tag to standard taxonomy (batch rename + taxonomy file update)

**Airflow Plugin API — Domain Terms** (`airflow/plugins/api/domain_terms.py`):

Mounted on the Airflow api-server at `/api/v1/domain-terms`. All endpoints use HMAC authentication and RLS tenant isolation via `rls_connection`.

- `POST /api/v1/domain-terms` -- Create a single domain term (201, 409 on duplicate)
- `POST /api/v1/domain-terms/bulk` -- Bulk create domain terms with `skip_duplicates` flag (201, 409, max 500)
- `PATCH /api/v1/domain-terms/{term_id}` -- Update domain term fields; cascades `term_original` rename to PG chunks and Milvus
- `DELETE /api/v1/domain-terms/{term_id}` -- Delete domain term; cascades removal from PG chunk JSONB and Milvus metadata (204, 404)

**Airflow Plugin API — Documents** (`airflow/plugins/api/upload.py`):

Mounted on the Airflow api-server at `/api/v1/documents`. HMAC authentication and RLS tenant isolation.

- `POST   /api/v1/documents/upload` -- Multipart upload from BFF; validates HMAC + extension + magic bytes + password-protection, writes to SeaweedFS, creates DB record in `pending_approval` state
- `POST   /api/v1/documents/{document_id}/approve` -- Approve a pending document; triggers `document_extraction_dag` via `api.dag.trigger_dag`
- `POST   /api/v1/documents/{document_id}/reject` -- Reject a pending document; deletes the staged S3 object
- `GET    /api/v1/documents/{document_id}/download-url` -- Presigned S3 URL with `attachment` content-disposition
- `GET    /api/v1/documents/{document_id}/preview-url` -- Presigned S3 URL with `inline` content-disposition (PDF preview)
- `DELETE /api/v1/documents/{document_id}` -- Delete document and all associated data (PG, Milvus, SeaweedFS)

**Airflow Plugin API — Block Tags** (`airflow/plugins/api/tags.py`):

Mounted on the Airflow api-server at `/api/v1/blocks`. HMAC authentication and RLS tenant isolation. Operates directly on the LlamaIndex `data_hierarchical_nodes` JSONB store and mirrors changes into Milvus chunk metadata via `upsert_chunk_field`.

- `PATCH  /api/v1/blocks/{chunk_id}/tags/status` -- Update a single tag's review state (`pending` / `verified` / `rejected`)
- `PATCH  /api/v1/blocks/{chunk_id}/tags/bulk-status` -- Batch state transitions for multiple tags on a chunk
- `PATCH  /api/v1/blocks/{chunk_id}/tags/{tag_id}/name` -- Rename a tag (1-255 chars)
- `DELETE /api/v1/blocks/{chunk_id}/tags/{tag_id}` -- Remove a tag from the chunk (204)

**Airflow Plugin API — Metrics** (`airflow/plugins/api/metrics.py`):

Prometheus instrumentation on the Airflow api-server. Counters/histograms registered on the global registry; relies on `AIRFLOW__API__WORKERS=1` (multi-worker scrape via `MultiProcessCollector` is out of scope).

- `GET /metrics` -- Prometheus scrape endpoint, HMAC-authenticated. Exposes upload/approve/reject/delete outcome counters, upload-duration histograms, and tag-endpoint duration/op counters.

### Semantic Tagger Pipeline

**SemanticTagger** (`src/extraction_v2/semantic_tagger.py`):
- LLM-based classification with config-driven taxonomy.
- Phase 2: ISO-compliant taxonomy with Hierarchy of Intent.
- Phase 3: Tag inheritance via `InheritanceEngine`, confidence scoring, JSONB metadata.
- Uses `PromptBuilder` to dynamically generate prompts from taxonomy config.
- Integrates `TagValidator` for pre-save validation.

### DomainAwareRetriever

- Implements Two-Phase Retrieval:
  1. **Pre-Filter:** Uses Domain Category and Terms to reduce search space.
  2. **Vector Search:** HNSW similarity search on filtered candidates.
- Manages multi-lingual domain term dictionary (En/Vi/Ja).

---

## LLM Provider Factory Architecture

> **C-REFRESH 2026-03-21:** Added based on code analysis of `src/llm/factory.py`.

The system uses a **provider-agnostic factory pattern** for all LLM operations:

```
┌─────────────────────────────────────────────────────────┐
│                    src/llm/factory.py                     │
├───────────────────┬─────────────────────────────────────┤
│  settings.llm_    │  settings.embedding_                │
│  provider          │  provider                           │
│  ┌─────┐ ┌──────┐ │  ┌─────┐ ┌──────┐                  │
│  │azure│ │ vllm │ │  │azure│ │ vllm │                  │
│  └──┬──┘ └──┬───┘ │  └──┬──┘ └──┬───┘                  │
│     │       │      │     │       │                      │
│  AzureOpenAI  OpenAI   │  AzureOpenAI  OpenAIEmbedding  │
│             (base_url)  │  Embedding   (api_base)        │
├─────────────────────────┴───────────────────────────────┤
│  Factory Functions:                                      │
│  • get_chat_client() / get_async_chat_client()          │
│  • get_chat_service()  (LangChain wrapper)              │
│  • get_embed_model()   (LlamaIndex)                     │
│  • get_async_embedding_client()                         │
│  • get_chat_model_name() / get_embedding_deployment()   │
├─────────────────────────────────────────────────────────┤
│  Fork-Safe HTTP Clients (httpx):                        │
│  • Sync: max_connections=1, no keepalive                │
│  • Async: max_connections=settings.llm_concurrency      │
└─────────────────────────────────────────────────────────┘
```

vLLM uses the standard OpenAI Python SDK with `base_url` pointed at the vLLM server, which exposes an OpenAI-compatible API. Provider selection is driven by `settings.llm_provider` and `settings.embedding_provider`.

---

## Extraction Module Inventory

> **C-REFRESH 2026-03-21:** Added based on code analysis.

### New Modules (not in original architecture doc)

| Module | Path | Purpose |
|--------|------|---------|
| `semantic_tagger.py` | `src/extraction_v2/` | LLM-based semantic tag classification with config-driven taxonomy, inheritance, confidence scoring |
| `domain_term_extractor.py` | `src/extraction_v2/` | Domain RAG term extraction with rule-based + LLM hybrid, CJK segmentation, configurable thresholds |
| `password_detection.py` | `src/extraction_v2/` | Early detection of password-protected Office docs (OOXML + OLE2) |
| `translation_service.py` | `src/extraction_v2/` | LLM-powered multi-lingual term translation (JA/EN/VI) |
| `inheritance_engine.py` | `src/extraction_v2/` | Tag inheritance cascading: document -> section -> block |
| `tag_validator.py` | `src/extraction_v2/` | Tag validation against Standard Taxonomy |
| `taxonomy_loader.py` | `src/extraction_v2/` | Taxonomy configuration loader with profile support |
| `tag_management_service.py` | `src/services/` | Tag CRUD operations with validation |
| `tag_review_service.py` | `src/services/` | HITL review workflow for pending tags |
| `tag_governance_service.py` | `src/services/` | Trending analysis, tag promotion, Shadow Taxonomy |
| `tags.py` | `src/api/routes/` | Tag management API routes (Phase 4) |
| `admin_taxonomy.py` | `src/api/routes/` | Admin taxonomy governance API routes (Phase 5) |
| `tags.py` | `src/api/schemas/` | Pydantic schemas for tags (TagAssignment, TagStatus, etc.) |
| `factory.py` | `src/llm/` | Provider-agnostic LLM factory (Azure/vLLM) |
| `chat_service.py` | `src/llm/` | LangChain ChatService wrapper |

---

## Data Architecture

### PostgreSQL Schema

```sql
-- Documents table
CREATE TABLE documents (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    filename VARCHAR(255) NOT NULL,
    file_path VARCHAR(500) NOT NULL,
    file_size_bytes INTEGER,
    status VARCHAR(50) DEFAULT 'pending',  -- pending, processing, completed, failed
    error_message TEXT,
    sheet_count INTEGER,
    record_count INTEGER,
    created_at TIMESTAMP DEFAULT NOW(),
    processed_at TIMESTAMP,
    api_key_id UUID REFERENCES api_keys(id)
);

-- ⚠️ Legacy code, deprecated — extracted_records, symbol_dictionaries, cross_references
-- These tables are written only by the legacy V1 pipeline (`save_extracted_records_task`,
-- `extraction/symbol_*.py`, `extraction/cross_ref_detector.py`) which is **not wired into
-- the active Airflow DAG**. The V2 Docling pipeline persists chunks via the dual-store
-- pattern (Milvus + PostgresDocumentStore) instead. Schema retained for historical
-- reference and any residual V1 deployments only.

-- Extracted records table (legacy V1)
CREATE TABLE extracted_records (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    sheet_name VARCHAR(255) NOT NULL,
    row_number INTEGER NOT NULL,
    col_range VARCHAR(50),  -- e.g., "A:F"
    content JSONB NOT NULL,  -- Original extracted content
    resolved_content JSONB,  -- After symbol resolution
    headers JSONB,  -- Column headers for this record
    tags JSONB DEFAULT '[]', -- Semantic tags (auto & manual) -- NOTE: JSONB not ARRAY
    created_at TIMESTAMP DEFAULT NOW(),

    CONSTRAINT unique_record UNIQUE (document_id, sheet_name, row_number)
);

-- Symbol dictionaries table (legacy V1)
CREATE TABLE symbol_dictionaries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    sheet_name VARCHAR(255),
    symbol VARCHAR(50) NOT NULL,
    meaning TEXT NOT NULL,
    context VARCHAR(255),
    created_at TIMESTAMP DEFAULT NOW()
);

-- Cross-references table (legacy V1)
CREATE TABLE cross_references (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    source_record_id UUID REFERENCES extracted_records(id) ON DELETE CASCADE,
    target_record_id UUID REFERENCES extracted_records(id) ON DELETE SET NULL,
    reference_text TEXT NOT NULL,
    reference_type VARCHAR(50),
    resolved BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT NOW()
);

-- ⚠️ Legacy code, deprecated — api_keys, query_logs
-- These tables back the legacy `src/api/` FastAPI surface (API-key auth +
-- /query log path). The active Airflow Plugin API uses HMAC + RLS instead;
-- query traffic has moved to chatbot-service / MCP search. The tables are
-- retained for historical reference until the cleanup PR lands.

-- API keys table (legacy V1)
CREATE TABLE api_keys (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    key_hash VARCHAR(255) NOT NULL UNIQUE,
    name VARCHAR(255),
    rate_limit_per_minute INTEGER DEFAULT 60,
    is_active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP DEFAULT NOW(),
    last_used_at TIMESTAMP
);

-- Query logs table (legacy V1)
CREATE TABLE query_logs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    api_key_id UUID REFERENCES api_keys(id),
    query_text TEXT NOT NULL,
    answer_text TEXT,
    confidence FLOAT,
    sources JSONB,
    processing_time_ms INTEGER,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Domain term dictionary table
CREATE TABLE domain_term_dictionary (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    term_original VARCHAR(255) NOT NULL UNIQUE,
    term_ja VARCHAR(255),
    term_en VARCHAR(255),
    term_vi VARCHAR(255),
    source_language VARCHAR(10) NOT NULL,
    domain_category VARCHAR(100) DEFAULT 'general',
    source_document_id UUID,
    translation_failed BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Indexes
CREATE INDEX idx_records_document ON extracted_records(document_id);
CREATE INDEX idx_records_sheet ON extracted_records(sheet_name);
CREATE INDEX idx_records_content ON extracted_records USING GIN (content);
CREATE INDEX idx_records_resolved ON extracted_records USING GIN (resolved_content);
CREATE INDEX idx_symbols_document ON symbol_dictionaries(document_id);
CREATE INDEX idx_symbols_symbol ON symbol_dictionaries(symbol);
CREATE INDEX idx_crossref_source ON cross_references(source_record_id);
CREATE INDEX idx_crossref_target ON cross_references(target_record_id);
CREATE INDEX idx_domain_dict_term_en ON domain_term_dictionary(term_en);
```

### Milvus Collection Schema

Milvus is managed by LlamaIndex's `MilvusVectorStore`. The collection stores **leaf nodes only** and is created with `enable_dynamic_field=True` — everything beyond the four declared columns lives in the per-row dynamic envelope (`$meta`).

```python
# Collection managed by LlamaIndex MilvusVectorStore
{
    "name": "<configured collection name>",
    "enable_dynamic_field": True,
    "schema": [
        {"name": "id",        "dtype": "VARCHAR",      "is_primary": True},
        {"name": "doc_id",    "dtype": "VARCHAR"},
        {"name": "text",      "dtype": "VARCHAR"},
        {"name": "embedding", "dtype": "FLOAT_VECTOR", "dim": 3072},
    ],
}
```

**Dynamic-envelope keys** (written by the ingest path via `node.metadata`; surfaced as top-level keys in query results):

| Key | Type | Source | Notes |
|---|---|---|---|
| `tenant_id`, `node_id` | str | Pipeline | Mandatory isolation keys (see `data-isolation-spec`). |
| `document_id`, `ref_doc_id`, `filename`, `document_type` | str | Pipeline | Document identity / provenance. |
| `page_number`, `total_pages`, `char_count`, `token_count` | int | Pipeline | Chunk-level metrics. |
| `schema_version` | str | `chunk_schema.CHUNK_SCHEMA_VERSION` | Currently `"1.1"`. |
| `tags` | list[dict] | `SemanticTagger` | Full `[{tag_id, name, status, confidence, ...}]` audit trail; not indexed, no app-side cap. |
| `verified_tag_names` | list[str] | `SemanticTagger._project_verified_tag_names` (ingest) + `airflow/plugins/api/milvus.project_verified_tag_names` (HITL re-projection) | Flat list of names where `status == "verified"`. Capped at 64 (warns on overflow). Cascade-filtered via scan-based `array_contains` / `array_contains_all` / `array_contains_any` — **no JSON-path INVERTED index** (TEXTIQ-569 iter 4). Pre-rollout chunks not touched by HITL lack the key; filters correctly no-match. |
| `domain_category`, `domain_terms` | str / list[str] | Domain-Aware RAG pipeline | Per `SPEC-DOMAIN-TERM`. |
| `_node_content`, `_node_type` | str | LlamaIndex internal | Serialized `TextNode` payload + class tag; required for hierarchical retrieval. |

**Embedding index:** declared by `MilvusVectorStore` at collection-create time (current default: `FLAT` + `IP`; switchable to `HNSW` + `COSINE` per deployment). Sparse `BM25` companion index lives on `sparse_embedding` for the BM25-augmented variant of the collection.

### PostgresDocumentStore Schema (LlamaIndex Hierarchical Nodes)

```python
# Table: hierarchical_nodes (managed by LlamaIndex PostgresDocumentStore)
# Namespace: "hierarchical"
# Storage: JSONB format (use_jsonb=True)
#
# PostgresDocumentStore.from_uri(
#     uri=database_url,
#     table_name="hierarchical_nodes",
#     namespace="hierarchical",
#     perform_setup=True,
#     use_jsonb=True,
# )
```

---

## API Contracts

> **Note:** All public REST traffic now hits the Airflow Plugin API (`/api/v1/*`) on the Airflow api-server. The `src/api/` FastAPI app and the contracts below are **legacy code, deprecated** and retained only for historical reference. See the "Airflow Plugin API — *" subsections under [Tagging & Governance Architecture](#tagging--governance-architecture-phase-4--5) for the active contracts.

### Document Upload — *legacy code, deprecated*

```http
POST /documents                      # superseded by POST /api/v1/documents/upload
Content-Type: multipart/form-data
X-API-Key: {api_key}
```

### Query — *legacy code, deprecated*

```http
POST /query                          # superseded by chatbot-service / MCP search
Content-Type: application/json
X-API-Key: {api_key}

{
  "query": "What does item code 2024-4_0019 appear in?",
  "document_ids": ["uuid1", "uuid2"],
  "include_context": true
}
```

### Tag Management (Phase 4) — *legacy code, deprecated*

```http
# Superseded by PATCH /api/v1/blocks/{chunk_id}/tags/{status,bulk-status,{tag_id}/name} and DELETE /{tag_id}
GET /v1/blocks/{record_id}/tags
POST /v1/blocks/{record_id}/tags
POST /v1/blocks/{record_id}/tags/review
```

### Admin Taxonomy (Phase 5) — *legacy code, deprecated*

```http
GET /v1/admin/taxonomy/trending?min_usage=50&min_documents=3&namespace=topic
GET /v1/admin/taxonomy/stats
POST /v1/admin/taxonomy/promote
```

### Health Check — *legacy code, deprecated*

```http
GET /health                          # superseded by Airflow api-server health
```

---

## Implementation Patterns

### Naming Conventions

| Entity | Convention | Example |
|--------|------------|---------|
| Python modules | snake_case | `table_detector.py` |
| Python classes | PascalCase | `TableDetector` |
| Python functions | snake_case | `detect_table_boundaries()` |
| Python constants | UPPER_SNAKE | `MAX_RETRIES = 3` |
| API endpoints | kebab-case | `/documents/{id}` |
| Database tables | snake_case plural | `extracted_records` |
| Database columns | snake_case | `created_at` |
| Environment vars | UPPER_SNAKE | `DATABASE_URL` |

### Error Handling

```python
class TextIQError(Exception):
    """Base exception for textiq-doc-extraction."""

class ExtractionError(TextIQError): pass
class RetrievalError(TextIQError): pass
class PasswordProtectedError(Exception): pass  # in extraction_v2/password_detection.py
```

---

## Security Architecture

### Authentication
- API key authentication via `X-API-Key` header
- Keys stored as SHA-256 hash in database
- Rate limiting per key (default: 60 requests/minute)

### Data Protection
- HTTPS only in production
- File uploads scanned for size limits
- No PII logging
- Document content not logged (only metadata)

---

## Performance Considerations

| Metric | Target |
|--------|--------|
| Document processing | <=60s per file |
| Query response | <=5s |
| Batch query (10) | <=30s |
| Concurrent queries | >=10 |

---

## Deployment Architecture

### Development
- Docker Compose with PostgreSQL, Redis, Milvus, SeaweedFS, Airflow
- uv for package management
- FastAPI on port 8000, Airflow on port 8080

### Production
- Airflow with CeleryExecutor or KubernetesExecutor
- FastAPI behind load balancer
- Managed PostgreSQL, Milvus Cluster, Redis Cluster
- Docker multi-stage builds with `/opt/airflow/.venv` managed by uv

---

## Architecture Decision Records (ADRs)

| ADR | Decision | Rationale |
|-----|----------|-----------|
| ADR-001 | Python over Node.js | Best ecosystem for data processing + LLM |
| ADR-002 | Separate Structured and Vector Stores | PostgreSQL for exact matches, Milvus for vectors |
| ADR-003 | Multi-Pass Extraction Pipeline | Symbol dictionaries must be resolved before indexing |
| ADR-004 | Airflow for Workflow Orchestration | DAG-based workflows, monitoring, SLA tracking |
| ADR-005 | Azure OpenAI for LLM | Enterprise compliance, GPT-4o capabilities |
| ADR-006 | Hybrid Excel Processing (Two Parallel Flows) | Large table efficiency + rich content extraction |
| ADR-007 | Hybrid Docling + openpyxl for Details Flow | openpyxl for visuals + Docling for parsing |
| ADR-008 | uv for Package Management | 10-100x faster than pip |
| ADR-009 | SeaweedFS for Object Storage (documents + images) | Distributed S3-compatible blob storage, self-hosted |
| ADR-010 | Domain Term Extraction | Improves semantic search for domain queries |
| ADR-011 | Mermaid-Aware 2-Level Chunking with LlamaIndex (TEXTIQ-625) | `MarkdownNodeParser` → branch-specific leaf splitter (text: `SentenceSplitter(512)`, Excel: `chunk_with_atomic_mermaid`). Dual-store: all nodes in PostgresDocumentStore for AutoMergingRetriever, leaves only in Milvus. Collapsed from the legacy 3-level shape on 2026-05-25. |
| ADR-012 | ks_xlsx_parser + markdownify for Excel Details (TEXTIQ-665) | ks_xlsx_parser handles merged cells and form-layout structures that Docling flattens; markdownify (lightweight, only bs4+six) replaces the Docling HTML→MD step; drops sentinel injection complexity (TEXTIQ-627) and eliminates color/cell-value metadata overhead. LibreOffice still handles `.xls`/`.xlsm` pre-conversion. |

---

## Deep-Dive Documentation

| Document | Scope | Description |
|----------|-------|-------------|
| [Airflow & Extraction V2 Integration](deep-dive-airflow-extraction-v2-integration.md) | `airflow/` <-> `src/extraction_v2/` | Complete data flow analysis, dependency graphs, and integration patterns |

---

## Change History

| Date | Version | Author | Changes |
|------|---------|--------|---------|
| 2026-02-03 | 1.0 | TextIQ | Initial architecture with hierarchical chunking |
| 2026-02-06 | 1.1 | TextIQ | Updated for dual-store architecture |
| 2026-03-21 | 2.0 (Blueprint) | nguyetnta8 | C-REFRESH: Added Tag Services (TagManagementService, TagReviewService, TagGovernanceService), API routes (tags.py, admin_taxonomy.py with trending/stats/promote), LLM Provider Factory (factory.py with Azure/vLLM), Extraction modules (semantic_tagger.py, password_detection.py, translation_service.py, inheritance_engine.py, domain_term_extractor.py, tag_validator.py, taxonomy_loader.py), Extraction Module Inventory table, updated project structure, updated FR mapping table, updated integration diagram. Added YAML frontmatter, TOC. |
| 2026-05-06 | 2.1 | nguyennh25 | Re-synced `airflow/` project structure with the implementation: corrected `dag_config.py`, dropped the non-existent `dags/operators.py` (operators live under `plugins/operators/`), added `chunk_schema.py`, listed all 14 `plugins/api/*` modules, added `plugins/operators/`, `plugins/models/xcom_schemas.py`, `airflow/config/` taxonomy assets, `airflow/scripts/`, and `docker/` build assets. Replaced the stub `Airflow Plugin API — Documents` section with the real `upload.py` endpoints (`/upload`, `/{id}/approve`, `/{id}/reject`, `/{id}/download-url`, `/{id}/preview-url`, `DELETE /{id}`), and added `Airflow Plugin API — Block Tags` (`/api/v1/blocks/{chunk_id}/tags/*`) and `Airflow Plugin API — Metrics` (`/metrics` Prometheus endpoint). Fixed FR-mapping rows for Document Management and Orchestration. |
| 2026-05-06 | 2.2 | nguyennh25 | Reconciled the project structure with the on-disk source tree and labelled every `src/` module with its liveness from the active Airflow code path. **(a) src/api FastAPI surface:** marked `src/api/{main.py, dependencies.py, middleware.py, routes/*}` as legacy code, deprecated — public REST traffic now goes through the Airflow Plugin API; annotated the project-structure block, the FR-mapping rows for API Access / Tag Management / Tag Governance, the "API Routes" subsection, and every entry under "API Contracts" (Document Upload, Query, Tag Management, Admin Taxonomy, Health Check). `src/api/schemas/*` is **not** deprecated — it remains in active use as a shared data-class library imported by `src/extraction_v2/{semantic_tagger,inheritance_engine}.py` and `src/services/tag_*_service.py`. **(b) Legacy V1 dead code:** flagged `src/extraction/` (V1 pipeline: `pipeline.py`, `excel_parser.py`, `table_detector.py`, `header_extractor.py`, `markdown_table_extractor.py`, `merged_cell_resolver.py`, `symbol_detector.py`, `symbol_resolver.py`, `cross_ref_detector.py`, `record_builder.py`, `models.py`) and `src/adapters/legacy_to_milvus_adapter.py` as legacy code, deprecated; annotated `airflow/dags/tasks/processing_tasks.py` to call out the orphaned legacy tasks (`chunk_text_task` L578-743, `generate_embeddings_task` L1275-1462, `generate_embeddings_chunks` L1463-1572, `merge_pipeline_results` L1573-1701, `store_vectors_task` L1855-2025, `save_extracted_records_task` L2148-2316); annotated `plugins/models/xcom_schemas.py` for the unused `ParsedDocument`, `ContentChunks`, and `ChunkEmbeddings` dataclasses (legacy code, deprecated). **(c) src/services/, knowledge/, retrieval/, qa/, llm/, models/, storage/ resync:** rewrote each block to match disk: `services/` now lists all 16 files with legacy/active tags (most are reached only via the legacy `src/api/routes/*`, except `pipeline_events`, `seaweedfs_service`, `azure_chat_service`, and the three tag services which are live independently); `knowledge/` adds `metadata_service`, `query_helper`, `term_index` and flags everything except `domain_dictionary_service` as legacy code, deprecated (only orphan-task consumers or zero importers); `retrieval/` corrected to the actual two files (`answer_generator`, `semantic_retriever`) and marked legacy code, deprecated (only via `/query` route); `qa/` and `models/` flagged as empty (only `__init__.py` — the previously listed files don't exist on disk); `llm/` corrected to the actual two files (`factory`, `chat_service`); removed the non-existent `src/storage/` block entirely. **(d) FR-mapping + PG schema:** rows for Symbol Resolution (FR11-14) and Cross-Reference (FR15-17) carry "legacy code, deprecated (V1, not in active DAG)" callouts; the `extracted_records`, `symbol_dictionaries`, and `cross_references` PG tables are flagged as legacy code, deprecated (written only by the unwired V1 pipeline). **(e) SeaweedFS clarification:** consolidated the Tech Stack rows into a single "Object Storage" entry, expanded the SeaweedFS bullet, and updated ADR-009 to make explicit that SeaweedFS is the S3-compatible blob store for **both** uploaded source documents (Excel, PDF, etc.) and extracted images. |
| 2026-05-06 | 2.3 | nguyennh25 | Flagged the `api_keys` and `query_logs` PG tables as legacy code, deprecated — they back the legacy `src/api/` FastAPI surface (API-key auth + `/query` log path). The active Airflow Plugin API uses HMAC + RLS instead, and query traffic has moved to chatbot-service / MCP search; the tables are retained only for historical reference until the cleanup PR lands. |
| 2026-05-13 | 2.4 | nguyennh25 | Resynced Milvus Collection Schema with the live `MilvusVectorStore` definition: declared four typed columns (`id`, `doc_id`, `text`, `embedding` dim=3072), `enable_dynamic_field=True`, and a per-key dynamic-envelope table covering `tenant_id`/`node_id`, document identity, chunk metrics, `schema_version` (= "1.1"), `tags`, the new `verified_tag_names` flat-array key from TEXTIQ-569 (cascade-filtered via scan-based `array_contains` predicates — no JSON-path INVERTED index per iter 4), `domain_category`/`domain_terms`, and the LlamaIndex `_node_content`/`_node_type` internals. Replaced the stale single-`metadata`-JSON sketch and the HNSW/COSINE claim with the real per-deployment index posture (`FLAT`+`IP` default with `HNSW`+`COSINE` switchable; sparse BM25 companion). |
| 2026-06-08 | 2.7 | nguyennh25 | TEXTIQ-665: Details flow now uses ks_xlsx_parser + markdownify `_FormattedConverter` instead of Docling. Dropped: font-style ATX sentinel injection (TEXTIQ-627 superseded for Details flow), `cell_colors`/`color_map`/`cell_values_map` from all chunk metadata, Docling `DocumentConverter` and `CellColorExtractor`. Kept: `_detect_and_extract_flowchart` (shapes→mermaid) + `collect_mermaid_blocks`/`restore_mermaid_blocks`. LibreOffice timeout bumped 60s→300s. Updated: Decision Summary (Excel Details row), Project Structure (added `ks_parser/` package), Details Flow step 1, Mermaid-Aware Chunking Goal A, embedding-budget defense point 4. Added: ADR-012. |
| 2026-05-26 | 2.6 | nguyennh25 | TEXTIQ-625 md-branch follow-up: aligned the `.md` sub-branch of `hierarchical_chunk_text_task` with the Excel branch — same `build_excel_parser_map` (`MermaidAwareLineSplitter(target=512, max=chunker_max)`) and same `chunker_max = embed_max - EMBED_SAFETY_BUFFER (400)`. Mermaid fences now stay inline with their explanatory prose (no orphan `node_type="mermaid_diagram"` leaves; no `[MERMAID_DIAGRAM_N]` placeholder). Architecture updates: refreshed the `chunk_text` step description in "Text Document Pipeline" (L579 area), the "Mermaid-Aware Chunking & Font-Style Header Injection" subsection (Goal B paragraph now states both Excel and `.md` share the splitter), and the "Chunk Sizes by Pipeline" table (split Text row into Word/PDF/PPTX vs `.md`, with `.md` matching the Excel splitter row). |
| 2026-05-25 | 2.5 | nguyennh25 | Synced architecture with three implemented specs (TEXTIQ-625 flowchart-mermaid pipeline, TEXTIQ-625 unified mermaid-aware chunking, TEXTIQ-627 xlsx font header injection). **(a) Decision Summary:** Chunking row rewritten from "3-level hierarchy [2048,512,256] / [4096,2048,1024]" to "`MarkdownNodeParser` + mermaid-aware splitter — 2-level: markdown → 512-token leaves (Excel uses `chunk_with_atomic_mermaid`)". **(b) Tech Stack — openpyxl:** dropped the "Flowchart detection and extraction" bullet; added the pre-Docling OOXML walk that materialises font-style ATX headers and serialises flowcharts as inline mermaid. **(c) Integration Points + Lifecycle Mermaid DAG:** updated `hierarchical_chunk_excel_detailed [4096,2048,1024]` → `chunk_excel_detailed` (markdown → `chunk_with_atomic_mermaid` target=512 / max≈7791) and `hierarchical_chunk_text [2048,512,256]` → `chunk_text` (markdown → `SentenceSplitter(512, "\\n\\n")`) in both the ASCII integration diagram and the mermaid lifecycle DAG. **(d) Details Flow + Text Pipeline narratives:** rewrote chunking step numbers, dropped flowchart-extraction language for the mermaid embedding path, added font-style ATX injection sentence and `MIN_CHUNK_TOKENS=64` / `prune_header_only_nodes` regex callouts. **(e) Hierarchical Chunking Architecture:** "Node Hierarchy" table compressed from 3 levels to 2 (parent = markdown section, leaf = 512); "Chunk Sizes by Pipeline" table rewritten for the 2-level shape with leaf-splitter column. **(f) Embedding Configuration:** added `EMBEDDING_MODEL_CONTEXT_TOKENS=8191`, `EMBED_SAFETY_BUFFER=400`, and `chunker_max=7791` rows. **(g) New subsection — Mermaid-Aware Chunking & Font-Style Header Injection:** documents Goal A (per-graph pre-Docling mermaid embedding), Goal B (`chunk_with_atomic_mermaid` line-based chunker), font-style sentinel scheme (`§§HN§§…§§/HN§§`), and the 4-layer embedding-budget defense (chunker cap, post-chunk audit, `EMBED_INCLUDE_KEYS` allow-list, parent-only `cell_colors`). |

---
_Updated 2026-06-08 -- TEXTIQ-665: Excel Details flow replaced with ks_xlsx_parser + markdownify; cell_colors/color_map dropped; sentinel injection removed._