---
title: 'TEXTIQ-389: domain-term confidence scoring (GAP-011)'
type: 'feature'
created: '2026-04-22'
status: 'complete'
baseline_commit: 'b73deaf'
context:
  - _planning/docs/planning-artifacts/2-planning/system-prd.md  # SYS-FR-47, SYS-FR-32
  - _planning/docs/services/textiq-doc-extraction/specs/extract_tag_function_spec.md  # Human=1.0
  - docs/implementation-artifacts/de-airflow-gap-audit-2026-04-20.md  # GAP-011
---

<frozen-after-approval reason="human-owned intent — do not modify unless human renegotiates">

## Intent

**Problem:** `domain_term_dictionary` has no `confidence` column, and the extraction pipeline never scores the terms it auto-adds. SYS-FR-47 requires each term to carry a 0–1 confidence so UPA can triage via the Green/Yellow/Red bands in SYS-FR-32.

**Approach:** Add `confidence FLOAT NOT NULL CHECK 0..1`. Compute it at every write site — rule-based match = **0.95**, LLM-extracted = **min of per-token probs** (weakest-link; from `logprobs=True` on the existing chat call), user-created via API = **1.0**. Fallback to 0.70 when logprobs unavailable/unalignable. No new GET endpoint; UPA still owns the HITL queue.

## Boundaries & Constraints

**Always:**
- `domain_term_dictionary.confidence`: `FLOAT NOT NULL`, `CHECK (confidence BETWEEN 0 AND 1)`.
- Rule-based match (`_extract_by_rules` hit) → **0.95**.
- LLM-extracted term → `exp(min(token_logprobs))` for the tokens spanning that term in the response, clamped to [0, 1]. (Weakest-link aggregation: one low-confidence token drags the whole term down.)
- User POST/bulk → **1.0** (tag-spec `Human = 1.0` precedent).
- Pass `logprobs=True, top_logprobs=0` on the chat.completions call (we only need the chosen-token logprob; alternatives not needed for scoring).
- If logprobs are missing (api-version too old, or provider returns None) OR the term text cannot be located in the token stream → fall back to **0.70**.
- If a term is produced by both rule and LLM paths, or by multiple chunks, take **max** across sources.
- Pydantic `DomainTermCreate.confidence: float | None` (ge=0, le=1); None defaults to 1.0 server-side.
- `persist_new_terms()` gains keyword-only `confidence_by_term: dict[str, float] | None`; when None, every row defaults to 0.70 (safe legacy fallback).
- Alembic migration reversible: upgrade backfills with 0.70 via temporary server_default then drops it; downgrade removes column + CHECK.

**Ask First:**
- Switching aggregation from min-prob to geometric mean (`exp(mean(logprobs))`) or arithmetic mean of probs (min-prob chosen for usable spread in the HITL Yellow band; geometric mean is the conventional length-normalized choice if calibration later demands it).

**Never:**
- No `GET /api/v1/domain-terms/` endpoint.
- No change to `data_hierarchical_nodes.value` chunk metadata shape.
- No change to Milvus schema.
- No prompt-format change to ask the LLM for per-term scores (future ticket).
- No per-term logprob verification pass (extra LLM call — future ticket if calibration proves insufficient).

## I/O & Edge-Case Matrix

| Scenario | Input / State | Expected | Error |
|---|---|---|---|
| POST omits `confidence` | `{term_original, source_language}` | 201; row + response `confidence=1.0` | — |
| POST sets `confidence=0.8` | valid | 201; stored 0.8 | — |
| POST sets `confidence=1.5` | out of range | 422 | — |
| Bulk mixed | 3 terms, two omit, one sets 0.5 | omitted→1.0, explicit→0.5 | — |
| DAG: rule-only hit | — | row 0.95 | — |
| DAG: LLM hit, logprobs clean | term tokens aligned | row = `exp(min(token_logprobs))` clamped | — |
| DAG: LLM hit, `logprobs` is None | old api-version / provider miss | row 0.70 (fallback) | warn-log once |
| DAG: LLM hit, term not found in token stream | strip_thinking / case drift | row 0.70 (fallback) | warn-log per term |
| DAG: same term from rules **and** LLM | both paths see it | row `max(0.95, llm_conf)` | — |
| DAG: term appears in N chunks | different logprobs each | keep `max` across chunks | — |
| Legacy caller, no dict | `persist_new_terms(terms=[...])` | all rows 0.70 | — |
| `alembic downgrade -1` | migration applied | column + CHECK gone | — |

</frozen-after-approval>

## Code Map

- `src/db/migrations/versions/20260422_100000_add_domain_term_confidence.py` — NEW; revises `4d5e6f7a8b9c`; adds `confidence FLOAT NOT NULL` + CHECK + backfill.
- `src/db/models.py:300` — add `confidence: Mapped[float]` on `DomainTermDictionary`.
- `src/extraction_v2/domain_term_extractor.py:466,577` — `_llm_extract` / `_llm_extract_async` call with `logprobs=True`; return `list[tuple[str, float]]` (term, confidence) instead of `list[str]`. New helper `_confidence_from_logprobs(response, terms)` that maps each parsed JSON term to its token span (via `(char_offset → token_idx)` mapping over concatenated `logprobs.content`) and returns `{term: exp(min(logprobs))}`. Fallback to 0.70 on any alignment failure.
- `src/extraction_v2/domain_term_extractor.py:628,725` — `extract_from_chunks_async` / `extract_from_nodes_async` return per-chunk `domain_confidences: dict[str, float]` alongside existing fields; rule-only terms get 0.95, LLM terms get their computed score, overlap takes max.
- `src/extraction_v2/domain_term_extractor.py:839` — `persist_new_terms` gains `confidence_by_term`; threads into `DomainTermDictionary(...)`.
- `airflow/dags/tasks/processing_tasks.py:795-860,1095-1124` — adapt both DAG call sites: aggregate `confidence_by_term` via max across chunks; forward to `persist_new_terms`.
- `airflow/plugins/api/domain_terms.py:44-56,113-258,317-397` — `DomainTermCreate.confidence` optional, INSERT writes it (default 1.0), responses echo.
- `tests/extraction_v2/test_domain_term_extractor.py` + integration — I/O matrix coverage including alignment edge cases.

## Tasks & Acceptance

**Execution:**
- [x] `src/db/migrations/versions/20260422_100000_add_domain_term_confidence.py` -- create migration revising `4d5e6f7a8b9c`; upgrade adds column w/ server_default 0.70 + CHECK, then drops default -- reversible backfill.
- [x] `src/db/models.py` -- add `confidence` column on `DomainTermDictionary`.
- [x] `src/extraction_v2/domain_term_extractor.py` -- pass `logprobs=True` on chat calls; implement `_confidence_from_logprobs(response, terms)`; change LLM extract return type; update async extractors to emit `domain_confidences` dict (rule=0.95, LLM=computed, max on overlap); add `confidence_by_term` kwarg to `persist_new_terms`.
- [x] `airflow/dags/tasks/processing_tasks.py` -- capture `domain_confidences` from both async extractors; aggregate across chunks via max; forward to `persist_new_terms`.
- [x] `airflow/plugins/api/domain_terms.py` -- add `confidence` to `DomainTermCreate`, INSERT it (default 1.0 when None), echo in all POST/bulk/PATCH responses.
- [x] Tests -- cover: user-omitted / user-in-range / user-out-of-range / bulk mixed / rule-only / LLM clean alignment / LLM logprobs=None fallback / LLM unalignable fallback / rule+LLM overlap max / legacy-caller default.

**Acceptance Criteria:**
- Given a fresh DB, when `alembic upgrade head` runs, then `domain_term_dictionary.confidence` is `NOT NULL FLOAT` with CHECK 0..1 and existing rows are 0.70.
- Given `alembic downgrade -1`, when applied, then column + CHECK are gone.
- Given an LLM extraction where `response.choices[0].logprobs is None`, when persisted, then every LLM-extracted term has confidence 0.70.
- Given an LLM-extracted term whose tokens are aligned in the response, when persisted, then its confidence equals `exp(min(token_logprobs))` clamped to [0, 1], within 1e-6.
- Given `persist_new_terms` called without `confidence_by_term`, when it completes, then every row is 0.70.

## Spec Change Log

<!-- populated by step-04 review loops -->

## Design Notes

**Aggregation = min of token probs (`exp(min(logprobs))`).** Chosen empirically over geometric mean: on a healthcare sample, geo-mean clustered 6/8 terms ≥0.97 (no usable spread for HITL Yellow band), while min-prob spread the same terms across 0.53–0.98. One low-confidence token drags the whole term down — this means legit multi-token terms sometimes land Yellow because the *first* token had ordering uncertainty (e.g., model was ~50/50 on which term to list first), but reviewers can approve those in seconds. Better than Green-band false confidence. Not calibrated in absolute terms; treat as ordinal signal for review ranking. Calibration is future work (UP-NFR-41).

**Fallback = 0.70, not fail-closed.** Logprob misalignment (thinking tokens, Unicode normalization drift, truncated JSON) is non-fatal; the term is still valid, we just lack a score. Yellow-band default keeps it in the review queue.

**Token↔term alignment.** Build concatenated text from `logprobs.content`, record `char_offset → token_idx`, `find(f'"{term}"')` on the concatenated text, slice token range, take `exp(min(logprobs))` over that slice. Handles `strip_thinking` implicitly (thinking chars precede the JSON — `find` skips past them).

## Verification

**Commands:**
- `alembic upgrade head && alembic downgrade -1 && alembic upgrade head` -- clean round-trip.
- `uv run pytest tests/extraction_v2/test_domain_term_extractor.py tests/integration -k "domain_term or confidence"` -- new cases pass, no regressions.
- `psql -c "\d domain_term_dictionary"` -- `confidence` NOT NULL with CHECK.

**Manual:**
- Trigger DAG on sample doc; `SELECT term_original, confidence FROM domain_term_dictionary ORDER BY created_at DESC LIMIT 20;` — distribution skewed high is expected (see Design Notes).
- HMAC-signed `POST /api/v1/domain-terms/` with and without `confidence`; response echoes value, row matches.
