# Deferred Work

Goals carved out of active specs. Revisit when the triggering need materializes.

## Embedding migration DAG (deferred from TEXTIQ-393 / GAP-039)

**Deferred on:** 2026-04-24
**Origin:** `spec-textiq-393-llm-provider.md`
**Original AC source:** `_planning/docs/planning-artifacts/2-planning/specs/llm-provider-abstraction-spec.md`

**What was cut:** An Airflow DAG (`embedding_migration_dag.py`) that reads chunks from `PostgresDocumentStore`, re-embeds via the configured provider, writes to a new Milvus collection, and atomically swaps via alias.

**Why deferred:** No concrete customer scenario needs it today. On-prem installs pick a model once and keep it; Azure↔on-prem boundary crossings re-ingest from source files. A mid-deployment model change is rare and currently addressable by a throwaway script. Building and maintaining a tested DAG for a hypothetical case is YAGNI.

**Re-activate when:** A real deployment swaps embedding model/dim mid-life with >1000 already-ingested documents where re-parsing from source would materially exceed re-embedding cost.

**Interim workaround:** Hand-write a `scripts/reembed_collection.py` for the one-off case. Read chunks from Postgres, call `get_embed_model()`, write to new Milvus collection, drop old.

**Ticket follow-up:** Amend TEXTIQ-393 AC3 — migration strategy = "re-ingest from source; dedicated migration tooling deferred." File a separate backlog ticket if/when the need surfaces.

## ~~Unify hierarchical chunking around markdown + extract mermaid atomically across all paths~~ — RESOLVED 2026-05-22

**Deferred on:** 2026-05-20
**Resolved on:** 2026-05-22 by `spec-textiq-625-unified-mermaid-aware-chunking.md` (commits up through `21d0aeb`).
**Origin:** `spec-textiq-625-flowchart-mermaid-pipeline.md` (carved off during step-01/02 to keep that spec ≤1600 tokens)

**What was cut originally:** Three changes in `airflow/dags/tasks/processing_tasks.py`:
1. Promote the md-branch mermaid handling (`r"```mermaid\b[^\n]*\n(.*?)\n```"` extraction, oversize-fence split along line boundaries against `Settings.embedding_model_context_tokens`, atomic `mermaid_diagram` TextNodes with `diagram_index`/`split_index`/`split_total`) into a shared helper.
2. Replace the `[4096, 1024, 256]` SentenceSplitter-only hierarchy in `hierarchical_chunk_text_task`'s non-md branch and in `hierarchical_chunk_excel_detailed_task` with the unified parser map.
3. Wire the mermaid helper in front of the hierarchy in all three branches so fences are handled correctly.

**How it landed (deviations from the original draft):**
- The shared helper module is `src/extraction_v2/mermaid_chunking.py`, providing `chunk_with_atomic_mermaid` (Excel-only, line-based mermaid-aware splitter), `split_oversize_mermaid`, `prune_header_only_nodes`, `build_excel_parser_map`, `build_text_parser_map`, and parser-id constants.
- Excel branch uses a 2-level hierarchy (`markdown → chunk_size_512`) with the line-based chunker — **in-context** mermaid (kept inside the prose chunk, not orphaned) — because Docling-xlsx output has no `\n\n` paragraph anchors and the original orphan-atomic design stripped diagrams from their introducing prose (`〔処理フロー〕` headers).
- md branch stays orphan-atomic for now (regex extracted to the helper, behavior preserved). Converting md to in-context mermaid is a separate follow-up.
- Non-md branch (Word/PDF/PPTX) gets the 3-level `MarkdownNodeParser → SS(1024, "\n\n") → SS(256, "\n\n")` hierarchy + `prune_header_only_nodes`, but **not** the line-based chunker (their paragraph structure is sane).
- Two-layer overflow defense added in commit `21d0aeb` after smoke-test surfaced (a) the embed metadata template eating into the 8192 budget and (b) `cell_colors` blowing the Milvus 64KB dynamic-field cap on highlight-heavy sheets.

**Carried forward as new deferred items:**
- md branch in-context mermaid migration (still orphan-atomic; covered by Spec Change Log entry in the unified spec).
- Font-style ATX header injection for xlsx (see entry below — independently confirmed today as the next biggest leverage point).

## Font-style-driven ATX header injection for Excel parsing

**Deferred on:** 2026-05-20
**Origin:** Prototype `scripts/inject_headers_and_parse.py` (built while validating `spec-textiq-625-flowchart-mermaid-pipeline.md`)

**What was cut:** A pre-Docling pass that walks `xl/styles.xml` + `xl/sharedStrings.xml` + `xl/worksheets/sheetN.xml`, classifies each cell by font (sz ≥ 14 bold → H1, bold ≥ body → H2, other bold → H3), and rewrites the cell text with sentinels (`§§HN§§…§§/HN§§`). After Docling, sentinels are converted to ATX headers (`#`, `##`, `###`), the `| … |` table wrappers around them are stripped, and adjacent duplicates from merged-cell unrolling are collapsed (H1 is deduped within-sheet since the merged title cell renders multiple times across rows/columns). Sibling cells inside any merge range containing a sentinel are cleared so Docling doesn't repeat the sentinel.

**Why this matters:** Docling's xlsx backend emits zero `#`/setext markdown headers — only pipe-tables. Result: `MarkdownNodeParser` finds nothing to split on and treats the whole document as ONE depth-0 section (52k tokens in our test file), so `HierarchicalNodeParser` degenerates to flat `SentenceSplitter` chunking that's blind to the doc structure. JP technical Excel specs encode hierarchy purely via font size/bold (e.g. `機能詳細説明書` sz=14, `〔処理フロー〕` sz=11 bold, `１．引数チェック` sz=9 bold). Injecting ATX headers from those font signals turned 1 depth-0 section into 89 (= 6 H1 + 36 H2 + 47 H3) on the sample file, matching the document's real semantic structure.

**Why deferred:** Outside the scope of `spec-textiq-625-flowchart-mermaid-pipeline.md` (that spec is about replacing the NL flowchart description with per-graph mermaid embedding; not about reshaping Docling's markdown output). Bundling header injection would double the spec's surface area, change semantics of `parse_to_markdown`, and require its own I/O matrix (font-size thresholds, body-size heuristic, merged-cell handling, sentinel collisions). It's independently shippable.

**Re-activate when:** Retrieval quality on JP Excel specs is measurably hurt by the flat depth-0 chunking (e.g. failure to scope step-level queries like "金融機関休業日フラグ判定 の処理" to the matching `### N．…` section). Likely soon after `spec-textiq-625-flowchart-mermaid-pipeline.md` + Goal-B (unified hierarchical chunking) land, since by then the markdown tier becomes the primary semantic carrier.

**Fresh evidence (2026-05-22 smoke run, post Goal-B landing):** After both `spec-textiq-625-flowchart-mermaid-pipeline.md` and `spec-textiq-625-unified-mermaid-aware-chunking.md` shipped, end-to-end test on `機能詳細説明書_…(IF).xls` (document_id `0be6fdf1-…`) produced 34 leaves in Milvus, of which **6 leaves are 18k–30k chars (6920 tokens at the top end)**. The largest, `chunk_id=390eb389-793c-42b7-8be1-0556579d5485`, covers one entire sheet (`BK20190220`, `start_char_idx=587`, `end_char_idx=30372`) as a single leaf because its `header_path` is `/` — the markdown contains zero `#`/`##`/`###` lines, so `MarkdownNodeParser` had nothing to split on and the section fell through whole to the line-based leaf chunker, which packed it up to the leaf budget. The chunker is now correct *given the input*; the input itself is the bottleneck. This is precisely the regression this deferred item exists to fix.

**Interim workaround:** None needed for correctness — chunks still contain the content and `AutoMergingRetriever` walks up to surface them; they're just not section-aligned, so per-section relevance is diluted on step-level queries.

**Ticket follow-up:** Create a spec `spec-xlsx-font-header-injection.md`. Carry forward from the prototype:
- Font classification heuristic + `--body-size` flag (default 11.0).
- Sentinel scheme (`§§HN§§…§§/HN§§`) survives Docling without escaping.
- Merge-range sibling clearing (otherwise the title-row spans 16-34 columns × 2 rows and yields duplicate sentinels).
- H1 override with sheet display name (the doc-title cell `機能詳細説明書` is identical across all sheets and useless as H1 otherwise).
- Within-sheet H1 dedup for non-adjacent merged-cell duplicates.
- Validation set: same xlsx samples used in `spec-textiq-625-flowchart-mermaid-pipeline.md`.