---
title: 'Unified Mermaid-Aware Hierarchical Chunking Across All DAG Branches'
type: 'refactor'
created: '2026-05-22'
status: 'ready-for-dev'
context:
  - docs/implementation-artifacts/deferred-work.md
  - docs/implementation-artifacts/spec-textiq-625-flowchart-mermaid-pipeline.md
---

<frozen-after-approval reason="human-owned intent — do not modify unless human renegotiates">

## Intent

**Problem:** Today the md branch of `hierarchical_chunk_text_task` is the only branch that handles ` ```mermaid ` fences, and it does so by **extracting them into orphan atomic leaves** — which removes the diagram from its surrounding prose context. The non-md branch (Word/PDF/PPTX) and `hierarchical_chunk_excel_detailed_task` use a flat `SS(4096) → SS(1024) → SS(256)` hierarchy with no mermaid handling, so the per-graph fences TEXTIQ-625 Goal A embeds into Excel sheets get sentence-split by the 256-token leaf splitter (verified: 10 of 12 mermaid leaves came out unbalanced on the IF workbook; 6 of 6 on the overview workbook). Separately, **Docling-produced Excel markdown is pathological** — overwhelmingly pipe-tables with very few `\n\n` paragraph breaks (overview: 0 ATX headers + 12 breaks across 377 lines; IF: 89 headers but 20k+-char unbroken table spans inside sections) — so `SentenceSplitter(paragraph_separator="\n\n")` falls through to char-level splits that bisect table rows mid-cell (prototype baseline: 40 mid-row leaves in IF). Word/PDF/PPTX and `.md` inputs do **not** share this pathology: their markdown has normal paragraph structure, so the standard `SentenceSplitter` on `\n\n` works correctly.

**Approach:** Add a shared helper `src/extraction_v2/mermaid_chunking.py` providing (a) a **line-based, mermaid-aware chunker** (`chunk_with_atomic_mermaid`) used **only by the Excel branch** to address the no-paragraph-break pathology, (b) a `prune_header_only_nodes` post-parse step (shared across branches), and (c) parser-config presets. The Excel chunker uses two budgets: pure-prose chunks stay under `chunk_size_target`; chunks that contain a mermaid fence may grow up to `embedding_model_context_tokens` so the fence stays adjacent to its explanatory prose. A single fence larger than the embedding context is split internally along its body lines. **Branch wiring:** Excel → 2-level (`MarkdownNodeParser → chunk_with_atomic_mermaid(target=512)`); non-md (Word/PDF/PPTX) and md → 2-level (`MarkdownNodeParser → SentenceSplitter(512, "\n\n")`) — leaf size aligned with the Excel target. The md branch's existing orphan-atomic mermaid extraction is **kept in place** in this spec (refactored only enough to share `prune_header_only_nodes`); migrating it to in-context mermaid is deferred so this spec stays Excel-focused. Prototype validation on both sample workbooks confirms Excel correctness: 0 unbalanced mermaid fences (was 16), 0 mid-row leaves (was 40), 0 header-only nodes (was 72), all mermaid in prose context.

## Boundaries & Constraints

**Always:**
- Helper lives in `src/extraction_v2/mermaid_chunking.py` and is importable from `@task.external_python` task bodies (same pattern as the existing `from src.core.config import Settings` import on line 460).
- The custom **line-based** chunker `chunk_with_atomic_mermaid` is used **only by `hierarchical_chunk_excel_detailed_task`**. Non-md and md branches use the stock LlamaIndex `SentenceSplitter` with `paragraph_separator="\n\n"`; their inputs have normal paragraph structure and don't need line-level splitting.
- In the Excel branch, mermaid fences are kept **in-context**: each `` ```mermaid…``` `` block stays inside the leaf chunk that contains its surrounding prose. No orphan atomic mermaid leaves are emitted from Excel; mermaid is never deleted from prose; no `node_type="mermaid_diagram"` metadata is set in Excel leaves.
- In the Excel branch, a chunk that contains at least one mermaid fence is exempt from the `chunk_size_target` budget and may grow up to `Settings().embedding_model_context_tokens`. Pure-prose chunks (no fence) respect `chunk_size_target` strictly.
- A single Excel-branch fence larger than `embedding_model_context_tokens` is split along its body lines; each part is re-wrapped in `` ```mermaid…``` ``. If a single body line alone exceeds the limit, `ValueError(f"single mermaid line exceeds embedding context ({n} tokens)")` is raised.
- After per-section chunking in the Excel branch, a backward-merge pass folds any chunk smaller than `MIN_CHUNK_TOKENS` (default 64) into the previous chunk **within the same section**, provided the combined size remains ≤ `embedding_model_context_tokens`. This eliminates tail-fragment leaves (e.g. a lone `| ⇒ | … |` table row left over after a flush boundary). Cross-section merging is never done; a small standalone section keeps its own leaf because mixing across section boundaries would corrupt semantic context.
- Excel branch parser ID list: `["markdown", "chunk_size_512"]` (2-level, leaf tier = `chunk_with_atomic_mermaid`). Non-md and md branches: `["markdown", "chunk_size_512"]` (2-level, leaf tier is a stock `SentenceSplitter(chunk_size=512, paragraph_separator="\n\n", chunk_overlap=20%)`). All branches now share the same `chunk_size_512` leaf id; `AutoMergingRetriever` walks each document's parent chain via the docstore.
- The md branch keeps its **existing orphan-atomic mermaid extraction** unchanged in behavior (the regex extract, oversize-fence line-split, and `TextNode` emission with `diagram_index`/`split_index`/`split_total`); only the regex and split helpers move into `mermaid_chunking.py` so they're shared with future callers. Converting the md branch to in-context mermaid is **deferred** (see Spec Change Log entry 5).
- After parsing, any node whose stripped text matches `^\s*#{1,6} [^\n]+\s*$` is dropped; parents whose entire subtree was dropped are removed recursively. Applied to all three branches. Validated on Excel IF: 24 header-only leaves + 48 header-only parents eliminated.

**Ask First:**
- Any change to the `chunk_size_target` numeric values (all branches now 512) — downstream `store_to_llamaindex_task` may key off these parser IDs.
- Removing the `chunk_sizes` kwarg from either task signature (rather than soft-deprecating it).
- Applying the line-based chunker to non-Excel branches — explicit out-of-scope for this spec (see "Never" below), but flag if a follow-up shows Word/PDF markdown also produces mid-paragraph fragments.

**Never:**
- Touch `_planning/` (planning-repo per CLAUDE.md).
- Use the line-based `chunk_with_atomic_mermaid` outside the Excel branch — Word/PDF/PPTX and `.md` inputs use stock `SentenceSplitter` on `\n\n`.
- Change md-branch mermaid behavior (it stays orphan-atomic in this release; conversion to in-context is a separate deferred follow-up).
- Backport the unified hierarchy to other call sites (`chunk_text_task`, `chunk_large_table_records`, `convert_large_tables_to_natural_language`) — out of scope.
- Inject ATX headers from xlsx font styles — separate deferred spec.
- Reshape image/flowchart-chunk handling in the Excel detailed task — those leaves remain flat TextNodes with their existing `element_type` metadata.
- Emit orphan TextNodes for mermaid from the Excel branch — Excel mermaid is in-context.

## I/O & Edge-Case Matrix

| Scenario | Input / State | Expected Output / Behavior | Error Handling |
|----------|--------------|---------------------------|----------------|
| Excel sheet w/ embedded mermaid + surrounding prose | Sheet markdown with N ` ```mermaid ` fences inline with table rows / headers | Each fence appears intact inside the leaf that contains its adjacent prose; leaf may exceed `chunk_size_target` but ≤ `embedding_model_context_tokens`; balanced fences; no orphan nodes | N/A |
| Excel mermaid-bearing leaf exceeds prose budget | Leaf has fence + prose totalling > target but ≤ context limit | Leaf kept whole; `has_mermaid=True` metadata; size reported in logs | N/A |
| Excel: oversize single mermaid fence | One fence > `embedding_model_context_tokens` | Fence split along inner body lines, each rewrapped in `` ```mermaid…``` ``, each part ≤ context limit | If one body line alone exceeds limit → `ValueError` |
| Excel: plain sheet (no mermaid) | Sheet markdown w/ tables only | All chunks ≤ `chunk_size_target=512`; line-based table-row splits; no leaf exceeds target | N/A |
| Excel: mermaid-only sheet | Sheet markdown is just one ` ```mermaid ` fence | One leaf containing the fence (and nothing else); `has_mermaid=True` | N/A |
| Excel: section whose body is just a header followed by sub-headers | E.g. `## A\n\n### A.1\n…\n### A.2\n…` produces an empty `## A` wrapper | Header-only node dropped; descendants kept; recursive prune removes parents whose entire subtree was header-only | N/A |
| Excel: tail-fragment leaf | Section ends with a sub-target chunk after a flush boundary (e.g. lone `| ⇒ | … |` row, 30 tok) | Folded backward into the prior chunk in the same section, ≤ embedding context | N/A |
| Non-md branch (Word/PDF/PPTX) | Multi-page Document with `page_number` metadata | 2-level hierarchy `MarkdownNodeParser → SentenceSplitter(512, "\n\n")`; `page_number` inherited from the markdown-section parent to the 512 leaf | Mermaid (if present) is split by SentenceSplitter as today — flagged as known-limitation; in-context preservation deferred |
| Md branch | Existing `.md` input | Same orphan-atomic mermaid behavior as today (regex extract + oversize line-split + atomic TextNodes with `diagram_index`/`split_index`/`split_total`); return-dict shape unchanged; header-only prune now applied | Same as before |
| Helper-level header-only prune | Mix of header-only and content nodes after parsing | Every node matching `HEADER_ONLY_RE` removed; ancestors whose entire subtree was pruned removed recursively | N/A |

</frozen-after-approval>

## Code Map

- `src/extraction_v2/mermaid_chunking.py` -- NEW. Public API: `MERMAID_FENCE_START_RE`, `FENCE_END_RE`, `HEADER_ONLY_RE`, `MIN_CHUNK_TOKENS`, `chunk_with_atomic_mermaid(text, target, max_size, tokenize) -> list[(str, bool)]` (Excel-only), `split_oversize_mermaid(block_lines, tokenize, limit) -> list[str]` (shared — also used by md branch's existing orphan path), `prune_header_only_nodes(nodes) -> list[BaseNode]` (shared across all branches), `EXCEL_PARSER_IDS = ["markdown", "chunk_size_512"]`, `TEXT_PARSER_IDS = ["markdown", "chunk_size_512"]`, `build_excel_parser_map(tokenizer, embed_ctx)`, `build_text_parser_map(tokenizer)` (returns `MarkdownNodeParser` + a single `SentenceSplitter(chunk_size=512, paragraph_separator="\n\n", chunk_overlap=20%)`).
- `airflow/dags/tasks/processing_tasks.py:399–638` -- md branch of `hierarchical_chunk_text_task`. **Behavior unchanged**; only refactor: import `MERMAID_FENCE_START_RE`/`FENCE_END_RE`/`split_oversize_mermaid` and `prune_header_only_nodes` from the helper instead of defining inline. Mermaid stays orphan-atomic in this release.
- `airflow/dags/tasks/processing_tasks.py:639–775` -- non-md branch of `hierarchical_chunk_text_task`. Replace flat `SS(4096) → SS(1024) → SS(256)` hierarchy with `build_text_parser_map` (`MarkdownNodeParser → SS(512, "\n\n")`); `page_number` inherited; apply `prune_header_only_nodes` post-parse.
- `airflow/dags/tasks/processing_tasks.py:2634–2853` -- `hierarchical_chunk_excel_detailed_task`. Replace flat SentenceSplitter hierarchy with the **line-based mermaid-aware chunker**: per-sheet `MarkdownNodeParser` → per-section `chunk_with_atomic_mermaid(target=512, max_size=embedding_model_context_tokens)` → backward-merge → `prune_header_only_nodes`; `sheet_name` inherited.
- `airflow/dags/document_extraction_dag.py:429,512` -- call sites; both pass defaults (no `chunk_sizes`). No code change, verified.
- `tests/extraction_v2/test_mermaid_chunking.py` -- NEW. Unit tests covering helper-level edge cases (Excel chunker, oversize raise, header-only prune, tail-merge).
- `tests/airflow/test_dags/test_document_extraction_dag.py:396` -- existing shape tests; verify they still pass.

## Tasks & Acceptance

**Execution:**
- [ ] `src/extraction_v2/mermaid_chunking.py` -- create with `chunk_with_atomic_mermaid` (Excel-only), `split_oversize_mermaid`, `prune_header_only_nodes`, `EXCEL_PARSER_IDS`, `TEXT_PARSER_IDS`, `build_excel_parser_map`, `build_text_parser_map` (stock SentenceSplitter on `\n\n`) -- shared logic for all three branches.
- [ ] `airflow/dags/tasks/processing_tasks.py` (md branch, ~451–638) -- refactor only: replace inline mermaid regex / split helper / header-only logic with helper imports; **mermaid stays orphan-atomic, behavior unchanged**.
- [ ] `airflow/dags/tasks/processing_tasks.py` (non-md branch, ~640–775) -- swap flat SentenceSplitter hierarchy for `build_text_parser_map` (`MarkdownNodeParser → SS(512, "\n\n")`); apply `prune_header_only_nodes`; `page_number` inherited.
- [ ] `airflow/dags/tasks/processing_tasks.py` (`hierarchical_chunk_excel_detailed_task`, ~2634–2853) -- swap flat SentenceSplitter hierarchy for **line-based mermaid-aware chunker** via `build_excel_parser_map` (2-level, in-context mermaid + tail-merge + header-only prune) -- closes the primary regression Goal A enabled.
- [ ] `tests/extraction_v2/test_mermaid_chunking.py` -- new pytest file covering the I/O Matrix scenarios (Excel chunker, oversize raise, header-only prune, tail-merge) -- locks behavior.
- [x] Smoke run: upload `【社外秘】【参考資料】ネットキャッシング対応_概要図.xls` via `scripts/test_upload_endpoint.py`, approve via `scripts/test_approval_endpoint.py`, poll DAG, query Milvus for `document_id == <id>` and verify (a) every leaf with `` ```mermaid `` in text has balanced fences, (b) total fences in leaves == fences detected by `extract_mermaid_graphs`, (c) no leaf body matches `HEADER_ONLY_RE` -- end-to-end gate. **PASS** on both `概要図.xls` (6/6 fences) and `IF.xls` (11/11 fences) -- see Spec Change Log entry on metadata-overflow fixes for the two-layer defense added during this run.

**Acceptance Criteria:**
- Given an Excel workbook with N embedded mermaid graphs across M sheets, when `hierarchical_chunk_excel_detailed_task` runs, then Milvus contains exactly N occurrences of `` ```mermaid `` across all leaves (counting split parts as 1 each when a fence had to be split), each occurrence balanced (` ```mermaid ` paired with closing ` ``` `), and every mermaid-bearing leaf carries `sheet_name`. No Excel leaf body matches `HEADER_ONLY_RE`. Every pure-prose Excel leaf is ≤ `chunk_size_target=512` tokens; mermaid-bearing leaves are ≤ `embedding_model_context_tokens`. No Excel leaf is < `MIN_CHUNK_TOKENS=64` unless it is a complete standalone section.
- Given a Word/PDF page, when `hierarchical_chunk_text_task` runs its non-md branch, then the document is chunked via `MarkdownNodeParser → SentenceSplitter(512, "\n\n")`; `page_number` is preserved on every leaf; no leaf body matches `HEADER_ONLY_RE`.
- Given an `.md` input identical to a known regression fixture, when the refactored md branch runs, then mermaid fence count in atomic leaves matches the pre-refactor `atomic_mermaid_count`; `node_count`/`leaf_count` unchanged (refactor is behavior-preserving); no leaf body matches `HEADER_ONLY_RE`.
- Given the existing `tests/airflow/test_dags/test_document_extraction_dag.py` suite, when re-run, all currently-passing tests still pass.

## Spec Change Log

- **2026-05-22**: Prototype validation on `scripts/inject_headers_and_parse.py` (initial: mermaid extracted to orphans + `paragraph_separator="\n"`) against both sample workbooks: mermaid fragments 16→0, prose leaves within budget. Spec adopted line-based splitting and orphan-atomic mermaid leaves.
- **2026-05-22**: Same prototype surfaced 24 header-only LEAVES + 48 header-only PARENTS. Added `prune_header_only_nodes` post-parse step; IF chunks 741→669, leaves 535→511, zero header-only nodes remain.
- **2026-05-22**: Compared 3-level (1024→256) vs 2-level (1024) vs 2-level (512) on IF. 2-level/512 produced 0 mid-row leaves, 0 oversize leaves, tight 53–444 token range. Excel branch adopted 2-level/512; non-md/md branches kept 3-level (1024→256) — branch-specific configs are compatible with `AutoMergingRetriever` because each document's parent chain is walked independently via the docstore.
- **2026-05-22**: **Replaced orphan-atomic mermaid with in-context mermaid (Excel only).** Reason: extracting mermaid into separate leaves stripped the diagram from its surrounding prose context (e.g. the `〔処理フロー〕` header that introduces the diagram ended up in a different leaf than the diagram itself, hurting retrieval). New design: mermaid stays in its prose chunk; the leaf containing it may grow past `chunk_size_target` up to `embedding_model_context_tokens`; only single fences exceeding the embedding context get split. Prototype on IF: 11/11 fences in context, all balanced, 0 mid-row leaves, max leaf 1146 tokens (well under 8191 cap).
- **2026-05-22**: Added backward tail-merge within sections to eliminate ≤64-token tail-fragment leaves left over after a flush boundary (e.g. lone `| ⇒ | … |` rows). IF final stats post-merge: 121 leaves, avg 391, median 401, min 55, max 1146, p95 584; zero leaves < 32 tokens. Cross-section merging never done — preserves semantic boundaries.
- **2026-05-22**: **Scoped the line-based chunker to the Excel branch only.** Reason: only Docling-xlsx markdown is pathological (no `\n\n` anchors → char-level splits bisect tables). Word/PDF/PPTX and `.md` inputs have normal paragraph structure and SentenceSplitter on `\n\n` works correctly there. md-branch mermaid handling stays orphan-atomic in this release (regex/helper extracted to the shared module for reuse, but behavior preserved); converting it to in-context mermaid is **deferred** as a separate follow-up so this spec stays focused on closing the Excel regression. Non-md branch adopts the 3-level `MarkdownNodeParser → SS(1024) → SS(256)` hierarchy on `\n\n` plus `prune_header_only_nodes`, but **does not** adopt the line-based chunker or in-context mermaid behavior.
- **2026-05-25**: **Non-md and md branches collapsed from 3-level to 2-level `MarkdownNodeParser → SentenceSplitter(512)`.** Reason: human-renegotiated after PDF smoke runs showed the 256-token leaves were over-fragmenting prose with no meaningful retrieval benefit from the 1024 mid-tier (AutoMerging rarely surfaced it in practice). Aligning the leaf size to 512 (same target as the Excel branch) cuts leaf count roughly in half and shortens DAG runtime ~30%. Updates: `TEXT_PARSER_IDS` → `["markdown", "chunk_size_512"]`; `build_text_parser_map` returns `MarkdownNodeParser` + single `SentenceSplitter(chunk_size=512, paragraph_separator="\n\n", chunk_overlap=int(512*0.2))`; md-branch inline parser map in `hierarchical_chunk_text_task` aligned to the same 2-level shape; `chunk_sizes` kwarg becomes a legacy-only knob ignored by both branches (warning still logged). Md-branch mermaid handling unchanged (still orphan-atomic — in-context conversion remains deferred). Tests `tests/extraction_v2/test_mermaid_chunking.py::TestParserMaps` updated to assert the 2-level shape; all 20 existing tests still pass. Verified end-to-end on `1706.03762v7 (Attention is All You Need).pdf`: 53 leaves (down from 114 on 3-level), all 15 pages covered, DAG `success` in ~3m02s (down from ~4m25s).
- **2026-05-22**: **Two-layer metadata-overflow defense added during IF-file smoke test.** Symptom 1: `openai.BadRequestError ... maximum input length is 8192 tokens` from the storage embed batch, even though `chunk_with_atomic_mermaid` was sizing leaves to `embedding_model_context_tokens` exactly. Root causes: (a) `leaf.get_content(MetadataMode.EMBED)` returns `metadata_template + text`, so the chunker's budget didn't account for the ~100-token template + filename/sheet_name suffix, plus Azure-tokenizer vs local `cl100k_base` drift on CJK content; (b) `store_to_llamaindex_task` attaches downstream metadata (`tags` — can be 600+ entries, `cell_colors`, `domain_terms`, `verified_tag_names`, `tenant_id`, `node_id`, `schema_version`, `header_path`, `has_mermaid`, `token_count`, `char_count`, `domain_category`) AFTER chunking, none of which the chunker's exclusion list covers. Fix: (1) `EMBED_SAFETY_BUFFER = 400` in `hierarchical_chunk_excel_detailed_task` — `chunker_max = embed_max_tokens - 400` is what `build_excel_parser_map` receives; (2) allow-list `EMBED_INCLUDE_KEYS = {filename, sheet_name, document_type, element_type}` in `store_to_llamaindex_task` — every other key is added to `excluded_embed_metadata_keys` + `excluded_llm_metadata_keys` at the last mutation point before embedding, so late-added metadata stays out of the embed input. Symptom 2: `MilvusException ... the length (196162) of dynamic field exceeds max length (65536)` after embedding succeeded. Root cause: `cell_colors` for a highlight-heavy sheet (12k+ colored cells ≈ 190KB JSON) was attached per-leaf; Milvus caps the dynamic-field envelope at 64KB per record. Fix: attach `cell_colors` only to non-leaf (parent / sheet-level) nodes — parents go to `PostgresDocumentStore` (no envelope cap) and `AutoMergingRetriever` walks up the parent chain to surface them; leaves stay slim for Milvus. `cell_values` removed entirely (duplicate of the markdown body, not read anywhere downstream). Added a per-leaf token audit in the chunking task (`MetadataMode.EMBED` measurement + top-5 size log) so future leaf-budget regressions surface in the chunking log instead of only at embed time. Verified end-to-end on both `概要図.xls` and `IF.xls` via `test_upload_endpoint.py` + `test_approval_endpoint.py`; smoke verifier `scripts/verify_mermaid_smoke.py` PASS on both (IF: 11/11 balanced fences across 6 leaves, 34 total leaves; overview: 6/6 balanced fences).

## Verification

**Commands:**
- `uv run pytest tests/extraction_v2/test_mermaid_chunking.py -v` -- expected: all helper tests pass.
- `uv run pytest tests/extraction_v2/test_excel_parser_flowchart.py tests/airflow/test_dags/test_document_extraction_dag.py -v` -- expected: existing suites still green.
- `uv run python scripts/test_upload_endpoint.py "【社外秘】【参考資料】ネットキャッシング対応_概要図.xls"` then `uv run python scripts/test_approval_endpoint.py approve <document_id>` -- expected: DAG `success`; Milvus query for `document_id==<id>` returns leaves where (a) every `` ```mermaid `` is balanced, (b) total mermaid count matches `extract_mermaid_graphs` count for the workbook (6 for this file), (c) no leaf body is a bare header line.

**Manual checks:**
- Inspect `parse_excel_detailed` task log: `"Flowchart mermaid embedded for N sheet(s)"` count matches the in-context mermaid count in Milvus for that document.
- Inspect `hierarchical_chunk_excel_detailed` task log: `mermaid_bearing_leaves` reported; histogram of leaf token sizes shows pure-prose leaves ≤ `chunk_size_target` and mermaid-bearing leaves ≤ `embedding_model_context_tokens`.