---
title: 'TEXTIQ-665 — Integrate ks_xlsx_parser into ExcelParserService'
type: 'feature'
created: '2026-05-29'
status: 'done'
baseline_commit: '99becfb0987cdefda6343d8f42bad2fee9d831e8'
context: []
---

<frozen-after-approval reason="human-owned intent — do not modify unless human renegotiates">

## Intent

**Problem:** `ExcelParserService` parses Excel via Docling's `DocumentConverter`, which fragments Japanese spec-style key-value forms and loses cell formatting. A hardened ks_xlsx_parser pipeline (form consolidation, bold/color injection, mermaid restore, header sentinels) already exists but lives in `scripts/inject_headers_and_parse_ks_fix.py` and runs only as a CLI.

**Approach:** Promote the ks pipeline into a production module under `src/extraction_v2/`, then route both service entry points (`parse_excel_file`, `parse_to_markdown`) through it: ks_xlsx_parser converts xlsx→HTML, markdownify (`_FormattedConverter`) converts HTML→MD preserving bold/color/strike inline. Keep image-description replacement, shape annotation preprocessing, and flowchart detection (STEP 1.5 converts drawing shapes→mermaid cells; ks then restores those fences). Drop service-side header-sentinel injection (ks materializes its own sentinels). Drop color/cell-value extraction entirely.

## Boundaries & Constraints

**Always:**
- ks_xlsx_parser is third-party/vendored — do NOT modify it; all fixes stay in the new adapter module.
- HTML→MD uses **markdownify** with a custom `_FormattedConverter` (subclass of `MarkdownConverter`) that preserves `<span style="color:...">` as inline HTML in the output; `<b>` → `**bold**`, `<s>` → `~~strikethrough~~` are handled natively. Do NOT use docling `export_to_markdown` for this step — it does not preserve inline style tags.
- Per-sheet output: group `ParseResult.chunks` by `chunk.sheet_name`; sheet order follows openpyxl `sheetnames`.
- Preserve existing public signatures of `parse_excel_file` and `parse_to_markdown` and their return shapes (`List[ChunkedElementDTO]` / dict with `sheets|image_chunks|flowchart_chunks`).
- Drop `color_map` and `cell_values_map` from both return shapes and from `ChunkedElementDTO` metadata — do not populate them.
- Image chunk creation stays unchanged.

**Ask First:**
- Deleting `scripts/inject_headers_and_parse_ks_fix.py` / `inject_headers_and_parse_ks.py` outright (default: leave a thin CLI shim that imports the new module).
- Deleting `_detect_and_extract_flowchart` and `_embed_mermaid_and_delete_shapes` from the service (default: keep — both methods are required for the mermaid two-stage pipeline).
- Adding a runtime feature flag (user chose hard replacement, not a toggle).

**Never:**
- Do not remove `self._convert_xls_to_xlsx` from the service — it is a pre-processing step (LibreOffice, timeout=300s) that runs before ks parses; ks only handles `.xlsx`.
- Do not remove `self.color_extractor = CellColorExtractor()` without also removing its import and the STEP 0/0.5 blocks that reference it.
- Do not keep `self.converter = DocumentConverter()` — it is removed (hard replacement, no toggle).
- Do not re-run header-sentinel injection (`inject_header_sentinels`) in the ks path — ks materializes its own sentinels via `_materialize_sentinels_safely`.
- Do not chunk in `parse_excel_file` — wrap each sheet's markdown as one DTO; chunking happens downstream.
- Do not change downstream DTO/metadata schema consumed by the md-ingestion DAG.
- Do not edit files under `_planning/`.

## I/O & Edge-Case Matrix

| Scenario | Input / State | Expected Output / Behavior | Error Handling |
|----------|--------------|---------------------------|----------------|
| Japanese KV form xlsx | `.xlsx` bytes | Per-sheet markdown with consolidated 2-col KV tables, bold/color spans, restored ```mermaid``` blocks | N/A |
| `.xls` / `.xlsm` input | legacy bytes | LibreOffice-converted to `.xlsx`, then ks pipeline | If LibreOffice missing → RuntimeError (existing behavior) |
| Workbook with mermaid flowchart cells | embedded ```mermaid``` | mermaid fences restored per sheet (multi-line, `-->` intact) | Unmatched block → warn, continue |
| Multi-sheet workbook | N sheets | N sheet entries; chunks grouped by `sheet_name` | Sheet with no chunks → skipped |
| Large sheet (parse_excel_file) | large sheet | Whole sheet markdown in one DTO — no splitting; chunking is downstream | N/A |
| ks_xlsx_parser raises | corrupt xlsx | Log error, return empty (`[]` / empty dict) — match current Docling-failure behavior | Caught, logged |

</frozen-after-approval>

## Code Map

- `scripts/inject_headers_and_parse_ks_fix.py` (1781 lines) -- SOURCE pipeline to promote (consolidation, `_extract_format`, `inject_format_tags_in_chunks`, `inject_strike_tags_in_chunks`, `_FormattedConverter` markdownify subclass, sentinel materialize, `ksparser_to_md`).
- `scripts/inject_headers_and_parse_ks.py` -- SOURCE of `collect_mermaid_blocks`, `restore_mermaid_blocks`, `convert_xls_to_xlsx`.
- `src/extraction_v2/excel_parser_service.py` -- integration target; both methods rewired.
- `src/extraction_v2/mermaid_flowchart_extractor.py` -- `_detect_and_extract_flowchart` KEPT as STEP 1.5 preprocessing (shapes→mermaid cells); `unwrap_mermaid_tables` call retired (no longer needed — ks restores fences).
- `src/extraction_v2/xlsx_header_injection.py` -- `inject_header_sentinels` + `convert_sentinels_to_atx` calls retired; ks does its own sentinel materialization.

## Tasks & Acceptance

**Execution:**
- [x] `src/extraction_v2/ks_parser/__init__.py` -- new package; export `ksparser_to_sheets`, `ksparser_to_md`.
- [x] `src/extraction_v2/ks_parser/pipeline.py` -- move pipeline + transforms from `inject_headers_and_parse_ks_fix.py`; add `ksparser_to_sheets(xlsx_path) -> List[{sheet_name, markdown}]` that groups `result.chunks` by `sheet_name`, builds per-sheet `html_body`, converts via `_FormattedConverter(heading_style="ATX").convert(html_body)` (markdownify — preserves inline span/bold/strike styles), then mermaid-restore + sentinel-materialize per sheet. Keep `ksparser_to_md` (whole-workbook) for the CLI. Fix module imports (no `sys.path` hacks; use package-relative).
- [x] `src/extraction_v2/ks_parser/mermaid.py` -- move `collect_mermaid_blocks`, `restore_mermaid_blocks`, `convert_xls_to_xlsx` from `inject_headers_and_parse_ks.py`.
- [x] `scripts/inject_headers_and_parse_ks_fix.py` -- reduce to thin CLI shim importing from `src.extraction_v2.ks_parser` (preserve `--converter` flag + output filename behavior).
- [x] `src/extraction_v2/excel_parser_service.py` (`parse_to_markdown`) -- remove STEP 0/0.5 (color/cell-value extraction), STEP 2.6 `inject_header_sentinels` call, STEP 4 Docling convert, STEP 5 MarkdownDocSerializer loop, `unwrap_mermaid_tables` call, `convert_sentinels_to_atx` call; **keep STEP 1.5 `_detect_and_extract_flowchart`** (converts drawing shapes → mermaid cells; ks then preserves those via `collect_mermaid_blocks`/`restore_mermaid_blocks`); replace STEP 4/5 with `ksparser_to_sheets(file_to_parse)`; drop `color_map`/`cell_values_map` from return dict (both success and failure paths); keep image-chunk STEP 6.
- [x] `src/extraction_v2/excel_parser_service.py` (`parse_excel_file`) -- remove STEP 0/0.5, STEP 3 `_extract_sheet_names`, STEP 4 Docling convert, STEP 5 `_chunk_by_sheet`; **keep STEP 1.5 `_detect_and_extract_flowchart`**; replace STEP 4/5 with `ksparser_to_sheets(file_to_parse)`; build `ChunkedElementDTO` directly from each `{sheet_name, markdown}` entry (one DTO per sheet, no splitting; do NOT route through `_convert_docling_chunks_to_dtos` — it expects Docling chunk objects); keep image-chunk STEP 6.
- [x] `src/extraction_v2/excel_parser_service.py` -- remove `self.converter = DocumentConverter()` and all docling imports that are no longer used; remove `self.color_extractor = CellColorExtractor()` and its import; remove dead methods: `_chunk_by_sheet`, `_convert_docling_chunks_to_dtos`.
- [x] `tests/` -- unit test ks adapter on a fixture xlsx: asserts per-sheet split, mermaid fences restored, bold/color spans present, separators normalized to `---` (covers I/O matrix rows).

**Acceptance Criteria:**
- Given a Japanese KV-form `.xlsx`, when `parse_to_markdown` runs, then returned `sheets` contain consolidated 2-column tables and restored ```mermaid``` blocks, and no service-side sentinel injection (`inject_header_sentinels`) executed.
- Given the same file, when `parse_excel_file` runs, then `ChunkedElementDTO`s carry correct `sheet_name` and ks-produced text; `cell_colors` and `cell_values` are absent from metadata; chunk count > 0.
- Given an `.xlsm` input, when either method runs, then LibreOffice conversion still precedes parsing.
- Given ks_xlsx_parser raising, when either method runs, then the method logs and returns empty without crashing the caller.
- Given the CLI shim, when `python scripts/inject_headers_and_parse_ks_fix.py <xlsx> --converter docling` runs, then output matches pre-refactor (11/11 mermaid).

## Verification

**Commands:**
- `uv run python scripts/inject_headers_and_parse_ks_fix.py "【社外秘】機能詳細説明書_ネットキャッシング振込予定年月日算出(IF).preprocessed.xlsx" --converter docling` -- expected: "restored mermaid newlines for 11/11", ~59k chars, no regression vs current.
- `uv run pytest tests/ -k ks_parser -q` -- expected: new adapter tests pass.
- `uv run python -c "from src.extraction_v2.ks_parser import ksparser_to_sheets"` -- expected: imports cleanly (no sys.path hacks).

## Design Notes

Per-sheet flow in `ksparser_to_sheets`:
```
mermaid_blocks = collect_mermaid_blocks(path)   # read xlsx once, before the sheet loop
chunks = parse_workbook(path).chunks             # carry .sheet_name, .render_html
chunks = reorder_chunks_by_page_columns(chunks, path)  # whole-workbook; needs cross-sheet page break context
by_sheet = group chunks by sheet_name (openpyxl order)

[per sheet:]
  → inject_strike_tags + inject_format_tags_in_chunks
  → consolidate_form_layout → merge_consecutive_form_pairs
  → split_chunks_by_row_gap → merge_sibling_column_chunks
  → normalize_chunks → filter empty
  → html_body = join(render_html for sheet's chunks)
  → md = _FormattedConverter(heading_style="ATX").convert(html_body)
  → md, _ = restore_mermaid_blocks(md, mermaid_blocks)  # reuse pre-collected blocks; non-match warns only
  → md, _ = _materialize_sentinels_safely(md)            # §§HN§§ → ATX
```

**Mermaid two-stage pipeline (both stages required):**
- STEP 1.5 `_detect_and_extract_flowchart` (service preprocessing): Excel drawing shapes (xdr:sp/xdr:cxnSp) → mermaid syntax → written into cells, shape XML deleted. Runs before `ksparser_to_sheets`.
- `collect_mermaid_blocks` / `restore_mermaid_blocks` (inside `ksparser_to_sheets`): reads cells that contain ` ```mermaid ` text (including those just written by STEP 1.5), then after HTML→MD conversion re-inserts properly fenced blocks. The two stages are complementary — dropping either loses mermaid output.