---
title: 'TEXTIQ-569 — Milvus: verified_tag_names flat-array column for fast tag filtering'
type: 'feature'
created: '2026-05-13'
status: 'done'
baseline_commit: 'f6d0fe7784c6ac565f4ccc32c286aef60f3fcb47'
context:
  - SPEC-SEMANTIC-TAG-RETRIEVAL v3.1 §2.4, §3.2, §6 (Chatbot)
  - SPEC-HITL-TAG (PATCH /blocks/{id}/tags/status)
---

<frozen-after-approval reason="human-owned intent — do not modify unless human renegotiates">

## Intent

**Problem:** Chatbot's tag-cascade filters (SPEC-SEMANTIC-TAG-RETRIEVAL v3.1 §3.2) match 0 rows against `tags` in `textiq_records` because the live collection has `enable_dynamic_field=True` and `tags` flows through the dynamic-JSON path as `list[dict]` (`{tag_id, name, status, ...}`). Filtering array-of-objects by nested key (`t.status == "verified"`) is not supported on the dynamic field — every cascade silently collapses to pure-vector fallback.

**Approach:** Add a derived, query-optimized flat array `verified_tag_names` stored as a **dynamic-field key** alongside the existing dynamic `tags`. Cascade filtering uses `array_contains` / `array_contains_all` / `array_contains_any` predicates on the dynamic key — scan-based, no JSON-path INVERTED index. Populate at ingestion (after the auto-approve gate) and re-project on every HITL mutation. ~~Backfill existing rows via partial upsert.~~ **Renegotiated 2026-05-13 (iter 3):** no backfill — new chunks receive the field at ingest, HITL mutations populate touched rows, and existing untouched chunks have no verified tags to begin with (cascade filter correctly no-matches missing keys). ~~JSON-path INVERTED index via `add_index`.~~ **Renegotiated 2026-05-13 (iter 4):** index removed — dynamic-field-only, scan-based filtering. ~~The live collection is altered in-place via `add_field` — no copy-swap.~~ **Renegotiated 2026-05-13 (iter 2):** dynamic-field path replaces the typed-column path; see Spec Change Log iteration 2.

## Boundaries & Constraints

**Always:**
- `verified_tag_names` contents = `[t.name for t in raw_tags if t.status == "verified"]`. Excludes `pending` and `rejected` (spec §2.4 — DE enforces the contract, not the consumer).
- Recompute the **whole array** on every write — never append/diff in place. Single source of truth for the projection is the post-mutation `tags` list.
- Existing dynamic `tags` field shape, contents, and (lack of) index are unchanged.
- Field definition: dynamic-field key `verified_tag_names` holding a JSON array of strings. Soft cap at 64 entries per chunk; values stored as JSON strings (no per-entry length limit enforced by schema). Existing rows untouched by HITL since this rollout have no key (missing ⇒ filter must not match — semantically correct: they have no verified tags).
- Filtering: scan-based `array_contains` / `array_contains_all` / `array_contains_any` against the dynamic key. No JSON-path INVERTED index — see Spec Change Log iter 4.
- Cap projection at `max_capacity=64`: log a warning and truncate (deterministic — preserve input order).
- Projection helper accepts `list[dict]` (HITL / dynamic-field path) and returns `[]` for any other shape (incl. legacy `list[str]` with no status info) — empty is the safe default; those rows re-populate on the next HITL mutation or re-ingest.

**Ask First:**
- ~~If the live Milvus server reports a version below 2.5.11, HALT and report the exact version before any index creation — JSON-path INVERTED on dynamic keys requires 2.5.11+.~~ **Renegotiated 2026-05-13 (iter 4):** index removed; no version gate applies.
- ~~If the backfill discovers any row whose `tags` field decodes to neither `list[str]` nor `list[dict]` (i.e. unknown shape), HALT and report sample row keys.~~ **Renegotiated 2026-05-13 (iter 3):** backfill removed entirely — gate no longer applies.
- ~~Before running the live `create_index` call, HALT and ask the human to schedule a low-traffic window (index build draws shared resources on a loaded collection).~~ **Renegotiated 2026-05-13 (iter 4):** no index DDL; gate moot.

**Never:**
- Do not change Chatbot code — that swap is a separate ticket.
- Do not modify how `tags` itself is stored in Milvus (still dynamic; out of scope).
- Do not disable `enable_dynamic_field` — other code paths still rely on it.
- ~~Do not store `verified_tag_names` as a dynamic field — defeats the ticket (dynamic JSON can't carry an INVERTED scalar index).~~ **Renegotiated 2026-05-13:** premise was outdated. Pivoted to dynamic-field storage (see iter 2). **Iter 4:** removed the index entirely — dynamic-field-only with scan-based `array_contains` filtering is sufficient for current collection size.
- ~~Do not auto-bump `CHUNK_SCHEMA_VERSION` — additive field; existing consumers continue to work without changes.~~ **Renegotiated 2026-05-13:** human chose to bump 1.0 → 1.1 (minor / additive) to surface the new column in the stamped envelope; see Spec Change Log.
- Do not silently drop verified tags above the cap; warn.

## I/O & Edge-Case Matrix

| Scenario | Input / State | Expected Behavior |
|----------|--------------|---------------------------|
| Fresh ingest, mixed statuses | 2 verified + 1 pending | row written with `verified_tag_names=["X","Y"]` (input order) |
| Fresh ingest, no verified | all pending or `tags=[]` | `verified_tag_names=[]` |
| HITL verify | `["X"]` → flip Y pending→verified | partial upsert `["X","Y"]` |
| HITL reject verified | `["X","Y"]` → flip Y verified→rejected | partial upsert `["X"]` |
| HITL rename verified | `["X","Y"]` → rename Y→Z | partial upsert `["X","Z"]` |
| HITL delete verified | `["X","Y"]` → delete Y | partial upsert `["X"]` |
| Projection overflow | 70 verified tags | truncate to 64; warn with `chunk_id` and `dropped=6` |
| Tagging task fails for a node | exception raised in `tag_records_task` | `node.metadata['verified_tag_names'] = []` set in the except branch (safety default) |
| Store path skips tagging | node arrives at `store_to_llamaindex_task` without the key | `setdefault('verified_tag_names', [])` ensures every new chunk carries the key |
| ~~Backfill: row absent in Milvus~~ | ~~non-leaf chunk~~ | **Renegotiated 2026-05-13 (iter 3): backfill removed.** Pre-rollout chunks not touched by HITL simply lack the key — semantically correct since they have no verified tags. |

</frozen-after-approval>

## Code Map

- `src/extraction_v2/semantic_tagger.py` -- `SemanticTagger._project_verified_tag_names(tags)` + `_stamp_verified_tag_names(records)`. Called immediately after `inheritance_engine.apply_inheritance` in both `tag_records` and `tag_records_async` so every record carries `record['verified_tag_names']`. Module-level constant `VERIFIED_TAG_NAMES_MAX_CAPACITY = 64`; over-cap truncates with a single `logger.warning("verified_tag_names_truncated", dropped=N)`.
- `airflow/dags/tasks/processing_tasks.py` -- (a) `tag_records_task` (Excel/Large-Table path): copies `record['verified_tag_names']` onto each chunk alongside `record['tags']`. (b) `tag_records_nodes_task` (hierarchical path): stamps `node.metadata['verified_tag_names']` from the tagger output. (c) `store_to_llamaindex_task`: `setdefault('verified_tag_names', [])` as safety default so every chunk carries the key. (d) `store_large_table_to_llamaindex_task`: reads `record.get('verified_tag_names', [])` at Document construction. LlamaIndex's `MilvusVectorStore` creates the collection with `enable_dynamic_field=True` and copies `node.metadata` keys into the Milvus dynamic envelope.
- `airflow/plugins/api/milvus.py:23` -- shared HITL helper `project_verified_tag_names` (unchanged) and `upsert_chunk_field` partial-upsert helper (writes into the dynamic envelope via `client.upsert(partial_update=True)`).
- `airflow/plugins/api/tags.py:190,287,354,406` -- four HITL call sites; each passes `verified_tag_names=...` alongside `tags=...`.
- `airflow/dags/tasks/chunk_schema.py` -- `CHUNK_SCHEMA_VERSION = "1.1"`; docstring covers top-level Milvus columns.
- ~~`scripts/add_verified_tag_names_field.py`~~ — **Removed in iter 4.** No index DDL needed; cascade filters use scan-based `array_contains` on the dynamic key.
- `src/knowledge/vector_store.py` -- **deprecated module, untouched.** The legacy `LegacyToMilvusAdapter` path that uses it is not the production ingest path.
- `tests/extraction_v2/test_semantic_tagger.py::TestVerifiedTagNamesProjection` (new), `tests/airflow/test_plugins/test_verified_tag_projection.py`, `tests/airflow/test_chunk_schema.py` -- unit-test the projection helpers, HITL endpoint wiring, and schema-version lock.

## Tasks & Acceptance

**Execution:**
- [x] `airflow/plugins/api/milvus.py` -- add `project_verified_tag_names(tags, chunk_id=None) -> list[str]`: accepts list[dict | obj] with `name`/`status` or legacy `list[str]`; filters `status=="verified"`; input order preserved; caps at 64 with `log.warning("verified_tag_names_truncated", chunk_id=..., dropped=N)` -- shared HITL projection
- [x] `airflow/plugins/api/tags.py` -- in `update_tag_status`, `bulk_update_tag_status`, `rename_tag`, `delete_tag`, change each call to `upsert_chunk_field(chunk_id, tags=tags, verified_tag_names=project_verified_tag_names(tags, chunk_id))` (use `updated_tags` in `delete_tag`) -- HITL re-projection
- [x] `src/extraction_v2/semantic_tagger.py` -- add `_project_verified_tag_names` + `_stamp_verified_tag_names` methods + `VERIFIED_TAG_NAMES_MAX_CAPACITY = 64` constant. Call `_stamp_verified_tag_names(records)` immediately after `apply_inheritance` returns, in both `tag_records` and `tag_records_async`. Single source of truth at ingest. -- ingest projection
- [x] `airflow/dags/tasks/processing_tasks.py` -- (a) `tag_records_task` (Excel/Large-Table): copy `record['verified_tag_names']` onto each chunk alongside `record['tags']`. (b) `tag_records_nodes_task` (hierarchical): stamp `node.metadata['verified_tag_names']` from `record.get('verified_tag_names', [])`; except branch defaults to `[]`. (c) `store_to_llamaindex_task`: in the metadata-stamping loop, `node.metadata.setdefault('verified_tag_names', [])` so every chunk carries the key. (d) `store_large_table_to_llamaindex_task`: read `record.get('verified_tag_names', [])` at Document construction. -- ingestion wiring via LlamaIndex dynamic-field
- ~~`scripts/add_verified_tag_names_field.py`~~ — **Removed in iter 4.** No index DDL; dynamic-field + scan-based `array_contains` filtering only.
- [x] `tests/extraction_v2/test_semantic_tagger.py::TestVerifiedTagNamesProjection` (7 cases) + `tests/airflow/test_plugins/test_verified_tag_projection.py` (18 cases) + `tests/airflow/test_chunk_schema.py` (2 cases) -- cover verified-only filter, empty inputs, ordering preserved, overflow capping + warning, mixed dict/object inputs, enum status, legacy `list[str]` → `[]`, stamp-every-record, **and** HITL endpoint wiring -- 27 passing
- [x] `airflow/dags/tasks/chunk_schema.py` -- `CHUNK_SCHEMA_VERSION` bumped `1.0` → `1.1` (additive — covers the new top-level dynamic-envelope key surfaced at ingest)

**Acceptance Criteria:**
- Given a fresh document ingestion where a chunk has 2 verified tags and 1 pending tag, when the Milvus row is queried for that chunk, then `verified_tag_names` contains exactly the 2 verified tag names in input order.
- Given a fresh document ingestion where tagging is skipped (e.g., empty text), when the Milvus row is queried for that chunk, then `verified_tag_names=[]` (set via `setdefault` in `store_to_llamaindex_task`).
- Given an existing row with `verified_tag_names=["X"]`, when `PATCH /blocks/{chunk_id}/tags/status` flips tag Y from `pending` to `verified`, then a subsequent `client.get(chunk_id)` returns `verified_tag_names=["X","Y"]`.
- Given an existing row with `verified_tag_names=["X","Y"]`, when `DELETE /blocks/{chunk_id}/tags/Y`, then the row's `verified_tag_names=["X"]`.
- Given a chunk with 70 verified tags, when written, then `verified_tag_names` length is 64 and a warning is logged with the chunk_id and `dropped=6`.
- Given a Milvus collection with `verified_tag_names` populated on a subset of rows, when a query expression `array_contains(verified_tag_names, "Foo")` is run, then matching rows are returned and rows missing the key no-match (no error).

## Spec Change Log

- 2026-05-13 — Bumped `CHUNK_SCHEMA_VERSION` 1.0 → 1.1 (minor / additive) at user request during implementation. Original spec said "Never auto-bump" because the constant tracks the metadata-envelope JSON, not top-level Milvus columns. User chose to bump so the additive change surfaces in `schema_version` stamps at ingest. Updated `tests/airflow/test_chunk_schema.py` lock test to assert `"1.1"`. KEEP: rest of Never list still applies (with iter-2/3 renegotiations).
- 2026-05-13 — Collection name: ticket text said `textiq_records_new` (matches `scripts/test_milvus_partial_update.py` hardcoded default), but live default per `airflow/plugins/api/milvus.py:23` is `textiq_records`. Spec and DDL-script default corrected to `textiq_records`.
- 2026-05-13 — **Iteration 2 (bad_spec loopback):** human discovered Milvus ≥ 2.5.11 supports JSON-path INVERTED indexing on dynamic-field keys (`index_type="INVERTED"`, `params={"json_cast_type":"varchar","json_path":"verified_tag_names"}`). Original spec premise — "dynamic JSON can't carry an INVERTED scalar index" — was outdated. Pivoted from typed-static-column to dynamic-field-with-JSON-path-INVERTED. Reverted: `MilvusEntity` typed columnar slot, `FieldSchema` in `vector_store.py`, typed `create_index`, `add_collection_field` in migration script. KEEP: projection helpers, four HITL endpoint wirings, `CHUNK_SCHEMA_VERSION="1.1"` bump, `--confirm` gate, idempotency log wording, tests for projection + HITL wiring.
- 2026-05-13 — **Iteration 3 (bad_spec loopback):** human flagged `src/knowledge/vector_store.py` as deprecated. The real ingest path is `airflow/dags/tasks/processing_tasks.py` — `store_to_llamaindex_task` / `store_large_table_to_llamaindex_task` use LlamaIndex's `MilvusVectorStore`, which creates the collection with `enable_dynamic_field=True` and copies `node.metadata` keys into the Milvus dynamic envelope. Pivot: revert all iter-2 edits to `src/knowledge/vector_store.py`. **Also removed backfill** ("we dont need backfill" — pre-rollout chunks without verified tags don't need the key; cascade filter correctly no-matches missing keys). **Projection placement (human-corrected mid-iteration):** initial attempt put the projection helper in a new `airflow/dags/tasks/tag_projection.py` module. Human redirected to extend `SemanticTagger` instead — `SemanticTagger` already owns the auto-approve gate that determines status, so emitting the verified-only flat array post-inheritance keeps both decisions in one place and yields fewer files. Added `_project_verified_tag_names` + `_stamp_verified_tag_names` methods to `SemanticTagger`, called immediately after `apply_inheritance` in both `tag_records` and `tag_records_async`. DAG tasks now consume `record['verified_tag_names']` directly (no DAG-side helper). Deleted: `airflow/dags/tasks/tag_projection.py` (and its tests), `tests/knowledge/test_verified_tag_projection.py`. Reduced: migration script is now index-DDL-only.
- 2026-05-13 — Review (step-04 iteration 1) patches applied — no intent_gap / bad_spec; ten findings classified `patch`, three `defer` (recorded in `deferred-work.md`), rest `reject`. Patches: (a) backfill `project()` warned + raised on unknown shape; (b) `--confirm` flag required for live runs; (c) idempotency log wording per AC; (d) `--dry-run` skips iterator when field absent; (e) `ensure_index` no longer swallows `MilvusException`; (f) Code Map + Verification updated; (g) frozen `Never: do not auto-bump CHUNK_SCHEMA_VERSION` annotated with renegotiation marker; (h) `chunk_schema.py` docstring widened; (i) overflow warning assertion. **Note (iter 3 supersession):** patches (a) and (d) — backfill-related — became moot when backfill was cut entirely. **Note (iter 4 supersession):** patches (b), (c), (e) — index-DDL-related — became moot when the script was removed entirely.
- 2026-05-13 — **Iteration 4 (human directive):** "remove the script, we only use dynamic field here and filter by array contain". Removed `scripts/add_verified_tag_names_field.py` entirely and dropped the JSON-path INVERTED index path. Cascade filtering uses scan-based `array_contains` / `array_contains_all` / `array_contains_any` predicates on the `verified_tag_names` dynamic key. Implications: no Milvus version floor specific to this ticket (dynamic-field support is pre-2.5 baseline); no operator step required at deploy time; query latency depends on collection size (acceptable for current scale, revisit if collection grows substantially). KEEP: projection helpers in `SemanticTagger` + `airflow/plugins/api/milvus.py`, four HITL endpoint wirings, ingest task stamps in `processing_tasks.py`, `CHUNK_SCHEMA_VERSION="1.1"` bump, all 27 tests. Removed: `scripts/add_verified_tag_names_field.py`, all server-version assertions, all `create_index`/`describe_index` references.

## Design Notes

**Dynamic field, no index.** Live `textiq_records` is created by LlamaIndex's `MilvusVectorStore` with `enable_dynamic_field=True`. The new column lives in the dynamic envelope alongside `tags` — no schema DDL, no typed `FieldSchema`, no JSON-path INVERTED index. Cascade filtering uses scan-based `array_contains` / `array_contains_all` / `array_contains_any` against the dynamic key. Acceptable at current scale; revisit indexing if query latency degrades as the collection grows.

**LlamaIndex carries metadata into the dynamic envelope.** `MilvusVectorStore.add(...)` calls `node_to_metadata_dict(node)`, which flattens `node.metadata` keys into the per-row entry written to Milvus. Any `node.metadata['verified_tag_names']` is therefore stored under the dynamic key of the same name. No special MilvusVectorStore configuration required.

**Projection placement.** The single source of truth for the ingest projection lives **inside `SemanticTagger`** — it already owns the auto-approve gate that determines status, so emitting the verified-only flat array post-inheritance keeps both decisions in one place. Every record returned by `tag_records` / `tag_records_async` carries `record['verified_tag_names']`. DAG tasks then copy it onto the chunk / node (no DAG-side projection helper needed). The HITL twin lives in `airflow/plugins/api/milvus.py` (boundary: plugin code cannot import from `src.*`) — ~15 lines of intentional duplication so each side owns its own projection.

**No backfill.** New chunks receive the field at ingest (via `setdefault` safety default in the store tasks). HITL mutations re-project on every write. Pre-rollout chunks not touched by HITL lack the key — semantically correct: they have no verified tags. The cascade filter (`array_contains(verified_tag_names, ...)`) on a missing key is no-match, which is the intended behaviour for those rows.

## Verification

**Commands:**
- `uv run pytest tests/extraction_v2/test_semantic_tagger.py::TestVerifiedTagNamesProjection tests/airflow/test_plugins/test_verified_tag_projection.py tests/airflow/test_chunk_schema.py -v` -- expected: all 27 cases pass
- `uv run python scripts/query_verified_tags.py --any --limit 5` -- expected: returns ingested chunks whose dynamic envelope carries `verified_tag_names` with at least one entry
- `uv run python scripts/check_milvus_schema_version.py <known_doc_id>` -- expected: a sample record's dynamic envelope contains `verified_tag_names` matching the verified subset of `tags`

**Manual checks:**
- Ingest a fresh document with mixed-status tags; query the resulting Milvus row and confirm `verified_tag_names ⊆ {t.name for t in tags if t.status=="verified"}` and the array is in input order.
- Trigger one HITL `PATCH .../tags/status` flip and re-query; confirm `verified_tag_names` reflects the change.
- In Attu: `array_contains(verified_tag_names, "<known_name>")` on the live collection returns expected rows; rows missing the key no-match (no error).
- Run a Chatbot smoke query that exercises the Strict-phase cascade against a tenant with known verified tags; confirm at least one row is returned (currently impossible per ticket; unblocks when Chatbot ticket lands).

## Suggested Review Order

**Shared projection contract (start here)**

- Ingest source of truth (`SemanticTagger._project_verified_tag_names` + `_stamp_verified_tag_names`): runs once after `apply_inheritance`; every returned record carries `record['verified_tag_names']`. Cap=64; warn on overflow.
  [`semantic_tagger.py` (search `_project_verified_tag_names`)](../../src/extraction_v2/semantic_tagger.py)

- HITL twin: `[t.name for t in tags if t.status=="verified"]`; cap at 64; warn with `chunk_id` + `dropped=N`. Intentional duplication — plugin code can't import `src.*`.
  [`milvus.py:23`](../../airflow/plugins/api/milvus.py#L23)

**Ingest wiring (LlamaIndex carries metadata → Milvus dynamic envelope)**

- `tag_records_task` (Excel/Large-Table) — copies `record['verified_tag_names']` onto each chunk alongside `record['tags']`.
  [`processing_tasks.py` `tag_records_task`](../../airflow/dags/tasks/processing_tasks.py)

- `tag_records_nodes_task` (hierarchical) — stamps `node.metadata['verified_tag_names'] = record.get('verified_tag_names', [])`. Except branch defaults to `[]`.
  [`processing_tasks.py` `tag_records_nodes_task`](../../airflow/dags/tasks/processing_tasks.py)

- `store_to_llamaindex_task` — in the metadata-stamping loop, `setdefault('verified_tag_names', [])` so every chunk carries the key even if tagging was skipped.
  [`processing_tasks.py` `store_to_llamaindex_task`](../../airflow/dags/tasks/processing_tasks.py)

- `store_large_table_to_llamaindex_task` — reads `record.get('verified_tag_names', [])` at Document construction.
  [`processing_tasks.py` `store_large_table_to_llamaindex_task`](../../airflow/dags/tasks/processing_tasks.py)

**HITL re-projection (4 endpoints — every tag-state mutation recomputes the whole array)**

- `PATCH /tags/status` — flip pending↔verified.
  [`tags.py:190`](../../airflow/plugins/api/tags.py#L190)

- `PATCH /tags/bulk-status` — bulk flips.
  [`tags.py:287`](../../airflow/plugins/api/tags.py#L287)

- `PATCH /tags/{id}/name` — rename.
  [`tags.py:354`](../../airflow/plugins/api/tags.py#L354)

- `DELETE /tags/{id}` — uses `updated_tags` (post-removal) for the projection.
  [`tags.py:406`](../../airflow/plugins/api/tags.py#L406)

**Schema-version stamp**

- `CHUNK_SCHEMA_VERSION` bumped to `"1.1"` (additive); docstring scope widened to top-level Milvus columns.
  [`chunk_schema.py:22`](../../airflow/dags/tasks/chunk_schema.py#L22)

**Tests (peripherals)**

- `SemanticTagger` projection: 7 cases — empty, dict-with-status, list[TagAssignment], legacy list[str], ordering preserved, overflow truncates+warns, `_stamp_writes_field_on_every_record`.
  [`test_semantic_tagger.py::TestVerifiedTagNamesProjection`](../../tests/extraction_v2/test_semantic_tagger.py)

- HITL projection + 4 endpoint wiring tests (18 cases).
  [`test_verified_tag_projection.py:1`](../../tests/airflow/test_plugins/test_verified_tag_projection.py#L1)

- Lock test pinned to `"1.1"` so any future bump shows up in the diff.
  [`test_chunk_schema.py:18`](../../tests/airflow/test_chunk_schema.py#L18)