# Prometheus Scraping — TextIQ DE Airflow Plugin

One scrape target exposes plugin metrics for the document-extraction service.
Airflow task-level metrics (DAG runs, scheduler heartbeats, etc.) are deferred
to GAP-029 Phase 2 — see TEXTIQ-391 ticket history for context.

## DE Plugin API server

Scrapes all three FastAPI sub-apps (shared global registry, single worker).

**Auth:** `/metrics` requires the same HMAC headers as every other plugin
endpoint (X-Tenant-ID, X-Node-ID, X-User-ID, X-Role, X-Timestamp, X-Signature).
Vanilla Prometheus cannot sign — its scrape config only supports
bearer/basic/mTLS/OAuth2. Two options:

1. **Internal scraper service** (preferred) — an in-trust-boundary TextIQ
   service that already has the `textiq_hmac` key ring signs and forwards
   metrics to wherever they need to go.
2. **Signing sidecar in front of Prometheus** — a small proxy that exposes
   plain `/metrics` to Prometheus and signs+forwards on each scrape. Not
   yet implemented; raise a ticket if needed.

```yaml
# Reference scrape config — assumes a signing intermediary is in front
scrape_configs:
  - job_name: textiq_de_plugin
    static_configs:
      - targets: ["<signing-proxy-host>:8081"]
    metrics_path: /api/v1/documents/metrics
```

Any of the three paths returns identical output (one global registry):
- `/api/v1/documents/metrics`
- `/api/v1/blocks/metrics`
- `/api/v1/domain-terms/metrics`

Key metrics:

| Metric | Type | Labels | Description |
|---|---|---|---|
| `upload_total` | counter | `outcome`, `tenant_id` | Document upload requests |
| `approve_total` | counter | `outcome`, `tenant_id` | Document approval requests |
| `reject_total` | counter | `outcome`, `tenant_id` | Document rejection requests |
| `delete_total` | counter | `outcome`, `tenant_id` | Document deletion requests |
| `upload_duration_seconds` | histogram | `outcome` | Upload handler latency |
| `hmac_verify_duration_seconds` | histogram | `result` | HMAC verification latency |
| `tag_op_total` | counter | `outcome`, `operation`, `tenant_id` | Tag mutation operations |
| `tag_endpoint_duration_seconds` | histogram | `operation` | Tag endpoint latency |
| `domain_term_op_total` | counter | `outcome`, `operation`, `tenant_id` | Domain-term mutation operations |
| `domain_term_endpoint_duration_seconds` | histogram | `operation` | Domain-term endpoint latency |

`outcome` values: `success` (2xx), `client_error` (4xx), `server_error` (5xx).

Single-worker note: `AIRFLOW__API__WORKERS=1` is required. If workers > 1 are ever
needed, migrate to `MultiProcessCollector` before scaling (separate spec required).