--- title: 'TEXTIQ-389: domain-term confidence scoring (GAP-011)' type: 'feature' created: '2026-04-22' status: 'complete' baseline_commit: 'b73deaf' context: - _planning/docs/planning-artifacts/2-planning/system-prd.md # SYS-FR-47, SYS-FR-32 - _planning/docs/services/textiq-doc-extraction/specs/extract_tag_function_spec.md # Human=1.0 - docs/implementation-artifacts/de-airflow-gap-audit-2026-04-20.md # GAP-011 --- ## Intent **Problem:** `domain_term_dictionary` has no `confidence` column, and the extraction pipeline never scores the terms it auto-adds. SYS-FR-47 requires each term to carry a 0–1 confidence so UPA can triage via the Green/Yellow/Red bands in SYS-FR-32. **Approach:** Add `confidence FLOAT NOT NULL CHECK 0..1`. Compute it at every write site — rule-based match = **0.95**, LLM-extracted = **min of per-token probs** (weakest-link; from `logprobs=True` on the existing chat call), user-created via API = **1.0**. Fallback to 0.70 when logprobs unavailable/unalignable. No new GET endpoint; UPA still owns the HITL queue. ## Boundaries & Constraints **Always:** - `domain_term_dictionary.confidence`: `FLOAT NOT NULL`, `CHECK (confidence BETWEEN 0 AND 1)`. - Rule-based match (`_extract_by_rules` hit) → **0.95**. - LLM-extracted term → `exp(min(token_logprobs))` for the tokens spanning that term in the response, clamped to [0, 1]. (Weakest-link aggregation: one low-confidence token drags the whole term down.) - User POST/bulk → **1.0** (tag-spec `Human = 1.0` precedent). - Pass `logprobs=True, top_logprobs=0` on the chat.completions call (we only need the chosen-token logprob; alternatives not needed for scoring). - If logprobs are missing (api-version too old, or provider returns None) OR the term text cannot be located in the token stream → fall back to **0.70**. - If a term is produced by both rule and LLM paths, or by multiple chunks, take **max** across sources. - Pydantic `DomainTermCreate.confidence: float | None` (ge=0, le=1); None defaults to 1.0 server-side. - `persist_new_terms()` gains keyword-only `confidence_by_term: dict[str, float] | None`; when None, every row defaults to 0.70 (safe legacy fallback). - Alembic migration reversible: upgrade backfills with 0.70 via temporary server_default then drops it; downgrade removes column + CHECK. **Ask First:** - Switching aggregation from min-prob to geometric mean (`exp(mean(logprobs))`) or arithmetic mean of probs (min-prob chosen for usable spread in the HITL Yellow band; geometric mean is the conventional length-normalized choice if calibration later demands it). **Never:** - No `GET /api/v1/domain-terms/` endpoint. - No change to `data_hierarchical_nodes.value` chunk metadata shape. - No change to Milvus schema. - No prompt-format change to ask the LLM for per-term scores (future ticket). - No per-term logprob verification pass (extra LLM call — future ticket if calibration proves insufficient). ## I/O & Edge-Case Matrix | Scenario | Input / State | Expected | Error | |---|---|---|---| | POST omits `confidence` | `{term_original, source_language}` | 201; row + response `confidence=1.0` | — | | POST sets `confidence=0.8` | valid | 201; stored 0.8 | — | | POST sets `confidence=1.5` | out of range | 422 | — | | Bulk mixed | 3 terms, two omit, one sets 0.5 | omitted→1.0, explicit→0.5 | — | | DAG: rule-only hit | — | row 0.95 | — | | DAG: LLM hit, logprobs clean | term tokens aligned | row = `exp(min(token_logprobs))` clamped | — | | DAG: LLM hit, `logprobs` is None | old api-version / provider miss | row 0.70 (fallback) | warn-log once | | DAG: LLM hit, term not found in token stream | strip_thinking / case drift | row 0.70 (fallback) | warn-log per term | | DAG: same term from rules **and** LLM | both paths see it | row `max(0.95, llm_conf)` | — | | DAG: term appears in N chunks | different logprobs each | keep `max` across chunks | — | | Legacy caller, no dict | `persist_new_terms(terms=[...])` | all rows 0.70 | — | | `alembic downgrade -1` | migration applied | column + CHECK gone | — | ## Code Map - `src/db/migrations/versions/20260422_100000_add_domain_term_confidence.py` — NEW; revises `4d5e6f7a8b9c`; adds `confidence FLOAT NOT NULL` + CHECK + backfill. - `src/db/models.py:300` — add `confidence: Mapped[float]` on `DomainTermDictionary`. - `src/extraction_v2/domain_term_extractor.py:466,577` — `_llm_extract` / `_llm_extract_async` call with `logprobs=True`; return `list[tuple[str, float]]` (term, confidence) instead of `list[str]`. New helper `_confidence_from_logprobs(response, terms)` that maps each parsed JSON term to its token span (via `(char_offset → token_idx)` mapping over concatenated `logprobs.content`) and returns `{term: exp(min(logprobs))}`. Fallback to 0.70 on any alignment failure. - `src/extraction_v2/domain_term_extractor.py:628,725` — `extract_from_chunks_async` / `extract_from_nodes_async` return per-chunk `domain_confidences: dict[str, float]` alongside existing fields; rule-only terms get 0.95, LLM terms get their computed score, overlap takes max. - `src/extraction_v2/domain_term_extractor.py:839` — `persist_new_terms` gains `confidence_by_term`; threads into `DomainTermDictionary(...)`. - `airflow/dags/tasks/processing_tasks.py:795-860,1095-1124` — adapt both DAG call sites: aggregate `confidence_by_term` via max across chunks; forward to `persist_new_terms`. - `airflow/plugins/api/domain_terms.py:44-56,113-258,317-397` — `DomainTermCreate.confidence` optional, INSERT writes it (default 1.0), responses echo. - `tests/extraction_v2/test_domain_term_extractor.py` + integration — I/O matrix coverage including alignment edge cases. ## Tasks & Acceptance **Execution:** - [x] `src/db/migrations/versions/20260422_100000_add_domain_term_confidence.py` -- create migration revising `4d5e6f7a8b9c`; upgrade adds column w/ server_default 0.70 + CHECK, then drops default -- reversible backfill. - [x] `src/db/models.py` -- add `confidence` column on `DomainTermDictionary`. - [x] `src/extraction_v2/domain_term_extractor.py` -- pass `logprobs=True` on chat calls; implement `_confidence_from_logprobs(response, terms)`; change LLM extract return type; update async extractors to emit `domain_confidences` dict (rule=0.95, LLM=computed, max on overlap); add `confidence_by_term` kwarg to `persist_new_terms`. - [x] `airflow/dags/tasks/processing_tasks.py` -- capture `domain_confidences` from both async extractors; aggregate across chunks via max; forward to `persist_new_terms`. - [x] `airflow/plugins/api/domain_terms.py` -- add `confidence` to `DomainTermCreate`, INSERT it (default 1.0 when None), echo in all POST/bulk/PATCH responses. - [x] Tests -- cover: user-omitted / user-in-range / user-out-of-range / bulk mixed / rule-only / LLM clean alignment / LLM logprobs=None fallback / LLM unalignable fallback / rule+LLM overlap max / legacy-caller default. **Acceptance Criteria:** - Given a fresh DB, when `alembic upgrade head` runs, then `domain_term_dictionary.confidence` is `NOT NULL FLOAT` with CHECK 0..1 and existing rows are 0.70. - Given `alembic downgrade -1`, when applied, then column + CHECK are gone. - Given an LLM extraction where `response.choices[0].logprobs is None`, when persisted, then every LLM-extracted term has confidence 0.70. - Given an LLM-extracted term whose tokens are aligned in the response, when persisted, then its confidence equals `exp(min(token_logprobs))` clamped to [0, 1], within 1e-6. - Given `persist_new_terms` called without `confidence_by_term`, when it completes, then every row is 0.70. ## Spec Change Log ## Design Notes **Aggregation = min of token probs (`exp(min(logprobs))`).** Chosen empirically over geometric mean: on a healthcare sample, geo-mean clustered 6/8 terms ≥0.97 (no usable spread for HITL Yellow band), while min-prob spread the same terms across 0.53–0.98. One low-confidence token drags the whole term down — this means legit multi-token terms sometimes land Yellow because the *first* token had ordering uncertainty (e.g., model was ~50/50 on which term to list first), but reviewers can approve those in seconds. Better than Green-band false confidence. Not calibrated in absolute terms; treat as ordinal signal for review ranking. Calibration is future work (UP-NFR-41). **Fallback = 0.70, not fail-closed.** Logprob misalignment (thinking tokens, Unicode normalization drift, truncated JSON) is non-fatal; the term is still valid, we just lack a score. Yellow-band default keeps it in the review queue. **Token↔term alignment.** Build concatenated text from `logprobs.content`, record `char_offset → token_idx`, `find(f'"{term}"')` on the concatenated text, slice token range, take `exp(min(logprobs))` over that slice. Handles `strip_thinking` implicitly (thinking chars precede the JSON — `find` skips past them). ## Verification **Commands:** - `alembic upgrade head && alembic downgrade -1 && alembic upgrade head` -- clean round-trip. - `uv run pytest tests/extraction_v2/test_domain_term_extractor.py tests/integration -k "domain_term or confidence"` -- new cases pass, no regressions. - `psql -c "\d domain_term_dictionary"` -- `confidence` NOT NULL with CHECK. **Manual:** - Trigger DAG on sample doc; `SELECT term_original, confidence FROM domain_term_dictionary ORDER BY created_at DESC LIMIT 20;` — distribution skewed high is expected (see Design Notes). - HMAC-signed `POST /api/v1/domain-terms/` with and without `confidence`; response echoes value, row matches.