# Phase 5: Tag Governance & Promotion Workflow

## Overview

Phase 5 implements the **Dynamic Tag Governance** system per Specification Section 4.4, enabling administrators to:
- Identify trending dynamic tags
- Promote popular dynamic tags to standard taxonomy
- Track tag usage statistics
- Maintain taxonomy evolution

## Architecture

### Components

1. **DynamicTagAnalyzer** - Analyzes tag usage across the database
2. **TagPromotionService** - Manages tag promotion lifecycle
3. **Admin API Endpoints** - RESTful interface for governance operations
4. **CLI Script** - Command-line tool for batch migrations

---

## Tag Lifecycle (Spec Section 4.4.1)

```
┌─────────────────────────────────────────────────────────────┐
│ Stage 1: Dynamic (Wild West)                                │
│ - User creates topic:gdpr                                   │
│ - Validation: Checks namespace only                         │
│ - Status: Valid Dynamic Tag                                 │
└─────────────────┬───────────────────────────────────────────┘
                  │ Usage grows...
                  ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 2: Candidate (Analysis)                               │
│ - System: Weekly batch job identifies trending tags         │
│ - Criteria: Usage > 50 blocks across 3+ projects            │
│ - Output: Taxonomy Dashboard alert for Admin                │
└─────────────────┬───────────────────────────────────────────┘
                  │ Admin reviews
                  ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 3: Promotion (Governance)                             │
│ - Action: Admin maps topic:gdpr → compliance:regulatory:gdpr│
│ - Decision: Add to standard or profile taxonomy             │
│ - Configuration: Update standard_tags.json or fintech.json  │
└─────────────────┬───────────────────────────────────────────┘
                  │ Execute
                  ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 4: Migration (Data Fix)                               │
│ - Batch Job: UPDATE all records with new tag name           │
│ - Re-indexing: Trigger vector embedding update (future)     │
│ - History: Add promotion entry to tag history                │
└─────────────────────────────────────────────────────────────┘
```

---

## API Endpoints

### 1. GET /v1/admin/taxonomy/trending

**Purpose:** Identify trending dynamic tags for promotion consideration

**Query Parameters:**
- `min_usage` (int, default: 50) - Minimum usage count
- `min_documents` (int, default: 3) - Minimum document count
- `namespace` (string, optional) - Filter by namespace (topic, entity, tech, domain)

**Response:**
```json
{
  "trending_tags": [
    {
      "name": "topic:gdpr",
      "usage_count": 127,
      "document_count": 8,
      "verified_ratio": 0.85,
      "first_seen": "2026-01-15",
      "last_seen": "2026-01-22"
    }
  ],
  "total_count": 15,
  "filters_applied": {
    "min_usage": 50,
    "min_documents": 3,
    "namespace": "topic"
  }
}
```

**Example:**
```bash
curl "http://localhost:8000/v1/admin/taxonomy/trending?namespace=topic&min_usage=50"
```

---

### 2. GET /v1/admin/taxonomy/stats

**Purpose:** Get overall tag usage statistics

**Response:**
```json
{
  "total_tags": 1523,
  "by_namespace": {
    "req": 456,
    "topic": 789,
    "compliance": 112,
    "entity": 166
  },
  "by_status": {
    "verified": 1200,
    "pending": 323
  },
  "dynamic_tags": 789
}
```

**Example:**
```bash
curl "http://localhost:8000/v1/admin/taxonomy/stats"
```

---

### 3. POST /v1/admin/taxonomy/promote

**Purpose:** Promote a dynamic tag to standard taxonomy

**Request Body:**
```json
{
  "old_tag": "topic:gdpr",
  "new_tag": "compliance:regulatory:gdpr",
  "description": "GDPR compliance requirements",
  "examples": [
    "Must comply with GDPR Article 17",
    "GDPR data retention policy"
  ],
  "profile": null,
  "add_to_taxonomy": true
}
```

**Parameters:**
- `old_tag` (required) - Current dynamic tag
- `new_tag` (required) - New canonical tag in standard namespace
- `description` (required if add_to_taxonomy=true) - Tag description
- `examples` (optional) - List of example usages
- `profile` (optional) - Profile to add to (fintech, healthcare, automotive)
- `add_to_taxonomy` (default: true) - Add to taxonomy config file

**Response:**
```json
{
  "success": true,
  "old_tag": "topic:gdpr",
  "new_tag": "compliance:regulatory:gdpr",
  "records_updated": 127,
  "added_to_taxonomy": true,
  "message": "Successfully promoted topic:gdpr to compliance:regulatory:gdpr. Updated 127 records. Added to taxonomy config."
}
```

**Example:**
```bash
curl -X POST "http://localhost:8000/v1/admin/taxonomy/promote" \
  -H "Content-Type: application/json" \
  -d '{
    "old_tag": "topic:gdpr",
    "new_tag": "compliance:regulatory:gdpr",
    "description": "GDPR compliance requirements",
    "examples": ["Must comply with GDPR Article 17"],
    "add_to_taxonomy": true
  }'
```

---

## CLI Script Usage

### List Trending Tags

```bash
# List all trending tags (usage >= 50)
python scripts/promote_dynamic_tag.py --list-trending

# Filter by namespace
python scripts/promote_dynamic_tag.py --list-trending --namespace topic

# Custom threshold
python scripts/promote_dynamic_tag.py --list-trending --min-usage 100
```

**Output:**
```
Trending Tags (usage >= 50):
================================================================================
Tag                                      Usage   Docs   Verified  First/Last Seen
--------------------------------------------------------------------------------
topic:gdpr                                 127      8      85.0%  2026-01-15 / 2026-01-22
topic:pci_dss                               89      5      92.1%  2026-01-12 / 2026-01-20
entity:stripe                               78      4      88.5%  2026-01-10 / 2026-01-22

Total: 3 trending tags
```

### Promote a Tag

```bash
# Promote to standard taxonomy
python scripts/promote_dynamic_tag.py \
  topic:gdpr \
  compliance:regulatory:gdpr \
  --description "GDPR compliance requirements" \
  --examples "Must comply with GDPR Article 17" "GDPR data retention"

# Promote to specific profile
python scripts/promote_dynamic_tag.py \
  topic:pci_dss \
  compliance:pci_dss \
  --description "PCI DSS compliance" \
  --profile fintech

# Database migration only (skip taxonomy file)
python scripts/promote_dynamic_tag.py \
  topic:authentication \
  req:quality:security \
  --no-taxonomy
```

**Output:**
```
Promoting tag:
  From: topic:gdpr
  To:   compliance:regulatory:gdpr

Step 1: Migrating database records...
✓ Successfully updated 127 records

Step 2: Adding to taxonomy configuration...
✓ Added to config/taxonomy/standard_tags.json

✓ Tag promotion complete!
  topic:gdpr → compliance:regulatory:gdpr
  Records updated: 127
```

---

## Workflow Example

### Scenario: Promoting "topic:gdpr" to Standard Taxonomy

**Step 1: Identify Trending Tags**

Admin runs weekly review:
```bash
curl "http://localhost:8000/v1/admin/taxonomy/trending?namespace=topic&min_usage=50"
```

Identifies `topic:gdpr` with 127 usages across 8 documents (85% verified ratio).

**Step 2: Review Tag Quality**

Admin checks tag usage in records:
```bash
curl "http://localhost:8000/v1/blocks/{record_id}/tags"
```

Confirms tag is used correctly and consistently.

**Step 3: Decide Canonical Mapping**

Admin maps `topic:gdpr` → `compliance:regulatory:gdpr` based on taxonomy structure.

**Step 4: Execute Promotion**

```bash
curl -X POST "http://localhost:8000/v1/admin/taxonomy/promote" \
  -H "Content-Type: application/json" \
  -d '{
    "old_tag": "topic:gdpr",
    "new_tag": "compliance:regulatory:gdpr",
    "description": "GDPR compliance requirements",
    "examples": ["Must comply with GDPR Article 17"],
    "add_to_taxonomy": true
  }'
```

**Step 5: Verify Migration**

- Database: All 127 records updated with new tag name
- History: Each tag has promotion entry added
- Config: `compliance:regulatory:gdpr` added to `standard_tags.json`
- Future uses: LLM now suggests `compliance:regulatory:gdpr` instead of `topic:gdpr`

---

## Database Migration Details

The promotion performs an **atomic batch update** using SQL:

```sql
UPDATE extracted_records
SET tags = (
    -- For each tag in the array
    SELECT jsonb_agg(
        CASE
            -- If tag matches old_tag, rename and add history
            WHEN tag->>'name' = 'topic:gdpr'
            THEN jsonb_set(
                jsonb_set(tag, '{name}', '"compliance:regulatory:gdpr"'),
                '{history}',
                tag->'history' || '[{"action": "promoted", "by": "admin", "ts": 1769067396, "old_value": "topic:gdpr"}]'
            )
            -- Otherwise, keep unchanged
            ELSE tag
        END
    )
    FROM jsonb_array_elements(tags) as tag
)
WHERE EXISTS (
    SELECT 1 FROM jsonb_array_elements(tags) as tag
    WHERE tag->>'name' = 'topic:gdpr'
);
```

**Result:**
- Tag name updated
- History entry added
- All metadata preserved (confidence, status, source)
- Transaction-safe (rollback on error)

---

## Taxonomy File Updates

When `add_to_taxonomy: true`, the tag is added to the appropriate config file:

### Standard Taxonomy (`config/taxonomy/standard_tags.json`)

```json
{
  "namespaces": {
    "compliance": {
      "tags": [
        {
          "name": "regulatory:gdpr",
          "description": "GDPR compliance requirements",
          "examples": [
            "Must comply with GDPR Article 17",
            "GDPR data retention policy"
          ]
        }
      ]
    }
  }
}
```

### Profile Taxonomy (`config/profiles/fintech.json`)

```json
{
  "additional_namespaces": {
    "compliance": {
      "tags": [
        {
          "name": "pci_dss",
          "description": "PCI DSS compliance",
          "examples": ["..."]
        }
      ]
    }
  }
}
```

---

## Best Practices

### 1. Review Before Promotion
- Check verified_ratio (aim for > 80%)
- Review actual usage in records
- Confirm tag is semantically correct

### 2. Choose Canonical Names Carefully
- Follow existing namespace patterns
- Use descriptive, unambiguous names
- Consider future taxonomy evolution

### 3. Run Weekly Analysis
```bash
# Cron job example (every Monday 9 AM)
0 9 * * 1 python scripts/promote_dynamic_tag.py --list-trending --min-usage 50 > /var/log/trending_tags.log
```

### 4. Document Decisions
- Add detailed descriptions
- Provide multiple examples
- Update internal wiki

### 5. Test Before Production
```bash
# Test on staging database first
python scripts/promote_dynamic_tag.py topic:test compliance:test --no-taxonomy
```

---

## Monitoring & Alerts

Set up alerts for:
- High trending tag count (> 20)
- Low verified ratios (< 60%)
- Rapid tag growth (usage spike)

Dashboard metrics:
```bash
curl "http://localhost:8000/v1/admin/taxonomy/stats"
```

---

## Troubleshooting

**Q: Tag promotion fails with "Tag not found"**
- Run `--list-trending` to verify tag exists
- Check exact spelling (case-sensitive)

**Q: Cannot add to taxonomy file**
- Check file permissions
- Verify JSON syntax in target file
- Ensure namespace exists in config

**Q: High number of pending tags**
- Review tagging quality with domain experts
- Adjust confidence thresholds
- Consider batch approval workflow

---

## Future Enhancements

- [ ] Auto-suggest canonical mappings (ML-based)
- [ ] Batch promotion (multiple tags at once)
- [ ] Tag merge (combine similar tags)
- [ ] Vector re-indexing trigger
- [ ] Taxonomy version control integration
- [ ] Rollback mechanism for promotions

---

## Related Documentation

- [Specification: Extract Tag Function](../specs/extract_tag_function_spec.md)
- [Phase 1: Data Model Migration](../migrations/)
- [Phase 2: Enhanced Taxonomy](../config/taxonomy/)
- [Phase 3: Inheritance & Confidence](../src/extraction_v2/)
- [Phase 4: HITL APIs](../src/api/routes/tags.py)
